00000800:00000100:2.1F:1787756042.370402:0:2359:0:(socklnd_cb.c:2294:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.201.118@tcp (192.168.201.118:988) the network or that node may be down. 00010000:00010000:3.0:1787756588.014761:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46975200/0x3252666880873ac8 lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c7992 expref: -99 pid: 177848 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014773:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014774:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014776:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb468c0200/0x3252666880873add lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb9:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c79c3 expref: -99 pid: 177849 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014784:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014785:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014787:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46994e00/0x3252666880873aa5 lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c7929 expref: -99 pid: 177846 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014793:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014794:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014796:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46974a00/0x3252666880873aba lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c795a expref: -99 pid: 177847 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014804:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014805:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014807:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46941800/0x3252666880873a82 lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb7:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c78c0 expref: -99 pid: 177844 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014814:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014815:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014817:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb4652a600/0x3252666880873a97 lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb7:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c78f1 expref: -99 pid: 177845 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014825:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014826:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014829:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb4697a400/0x3252666880873a58 lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c7842 expref: -99 pid: 177842 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014851:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014852:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014854:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46940c00/0x3252666880873a74 lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb6:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c7888 expref: -99 pid: 177843 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014862:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014863:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014865:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46872600/0x32526668808739fd lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c77c4 expref: -99 pid: 177838 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014872:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014873:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014875:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46872c00/0x3252666880873a4a lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb5:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c780a expref: -99 pid: 177839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014882:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014883:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014886:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb4652b800/0x32526668808739da lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c775b expref: -99 pid: 177836 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014895:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014897:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014899:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46873e00/0x32526668808739ef lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c778c expref: -99 pid: 177837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014908:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014909:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014912:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46872800/0x32526668808739b7 lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c76f2 expref: -99 pid: 177834 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014924:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014925:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014927:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46873a00/0x32526668808739cc lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb3:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c7723 expref: -99 pid: 177835 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014935:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014936:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014938:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46975c00/0x325266688087398d lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c7674 expref: -99 pid: 177832 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014944:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014945:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014947:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46941400/0x32526668808739a9 lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb2:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c76ba expref: -99 pid: 177833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014954:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014955:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014957:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46873000/0x3252666880873932 lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c75f6 expref: -99 pid: 177828 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014963:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014964:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014966:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46940e00/0x325266688087397f lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb1:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c763c expref: -99 pid: 177829 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.014974:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014975:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014977:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46940400/0x325266688087390f lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eb0:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c758d expref: -99 pid: 177826 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.014992:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.014993:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.014996:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46872400/0x3252666880873924 lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eb0:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c75be expref: -99 pid: 177827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.015005:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.015006:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.015008:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46872200/0x32526668808738ec lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eaf:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c7524 expref: -99 pid: 177824 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.015014:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.015016:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.015018:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46873600/0x3252666880873901 lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eaf:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c7555 expref: -99 pid: 177825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.015026:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.015027:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.015030:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb467ede00/0x32526668808738c9 lrc: 0/0,0 mode: --/CR res: [0x2000032e1:0x1eae:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xbc9d070b057c74bb expref: -99 pid: 177822 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00000001:3.0:1787756588.015045:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.015046:0:152733:0:(ldlm_lock.c:185:ldlm_lock_put()) Process entered 00010000:00010000:3.0:1787756588.015048:0:152733:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ffb481c8000 lock: ffff8ffb46872a00/0x32526668808738de lrc: 0/0,0 mode: --/PR res: [0x2000032e1:0x1eae:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xbc9d070b057c74ec expref: -99 pid: 177823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00000001:3.0:1787756588.015057:0:152733:0:(ldlm_lock.c:224:ldlm_lock_put()) Process leaving 00010000:00000001:3.0:1787756588.015058:0:152733:0:(ldlm_request.c:2504:ldlm_cli_cancel_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1787756588.015062:0:152733:0:(ldlm_lockd.c:2923:ldlm_bl_thread_blwi()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756588.015066:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756588.015068:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756588.015070:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756588.015072:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756588.015074:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756588.015076:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756588.015078:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756588.015079:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756588.015081:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756588.801367:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756588.801375:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756588.801378:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756588.801385:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756588.801390:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756588.801394:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756588.801397:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756588.801400:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756588.801403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756589.825367:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756589.825379:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756589.825384:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756589.825393:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756589.825400:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756589.825407:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756589.825412:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756589.825416:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756589.825421:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756590.273294:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756590.273306:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756590.273309:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756590.273313:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756590.273318:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756590.273326:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9645 00000100:00080000:3.0:1787756590.273338:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756590.273342:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756590.273345:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756590.273349:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756590.273353:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756590.273357:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756590.273360:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756590.273363:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756590.273364:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756590.273369:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756590.273372:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756590.273375:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756590.273379:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756590.273382:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273385:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756590.273388:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756590.273401:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756590.273403:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273407:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756590.273410:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756590.273415:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273456:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756590.273468:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659563136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756590.273481:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756590.273486:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273505:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756590.273509:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756590.273513:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756590.273515:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756590.273517:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756590.273520:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756590.273523:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9645 00000100:00080000:3.0:1787756590.273527:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756590.273529:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756590.273531:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756590.273533:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756590.273536:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756590.273539:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756590.273542:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756590.273545:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756590.273547:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756590.273551:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756590.273554:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756590.273557:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756590.273564:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756590.273567:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273571:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756590.273577:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756590.273584:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273588:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756590.273596:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756590.273601:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273606:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756590.273609:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756590.273617:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273632:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756590.273644:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659563264/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756590.273664:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756590.273675:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273686:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756590.273691:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756590.273697:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756590.273699:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756590.273702:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756590.273712:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756590.273715:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9645 00000100:00080000:3.0:1787756590.273719:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756590.273722:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756590.273724:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756590.273726:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756590.273728:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756590.273730:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756590.273734:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756590.273743:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756590.273746:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756590.273756:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756590.273760:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756590.273766:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756590.273769:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756590.273772:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273774:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756590.273776:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756590.273780:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273782:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756590.273805:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756590.273807:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273809:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756590.273811:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756590.273814:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273820:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756590.273826:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659563392/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756590.273862:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756590.273865:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.273869:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756590.273873:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756590.273883:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756590.273888:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756590.273895:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756590.273901:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756590.273910:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9645 00000100:00080000:3.0:1787756590.273960:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756590.273965:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756590.273968:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756590.273976:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756590.273980:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756590.273982:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756590.273985:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756590.273988:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756590.273990:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756590.273995:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756590.274003:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756590.274006:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756590.274010:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756590.274017:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.274021:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756590.274026:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756590.274031:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.274039:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756590.274046:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756590.274051:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.274057:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756590.274059:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756590.274061:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.274070:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756590.274082:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659563520/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756590.274122:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756590.274127:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.274131:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756590.849318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756590.849330:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756590.849334:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756590.849342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756590.849348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756590.849353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756590.849356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756590.849360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756590.849363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756591.873348:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756591.873360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756591.873362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756591.873370:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756591.873376:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756591.873381:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756591.873385:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756591.873388:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756591.873438:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756592.065571:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756592.065579:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756592.065585:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756592.897296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756592.897304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756592.897307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756592.897312:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756592.897316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756592.897320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756592.897323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756592.897326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756592.897329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756593.921428:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756593.921439:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756593.921442:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756593.921451:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756593.921456:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756593.921462:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756593.921465:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756593.921469:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756593.921472:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756594.945301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756594.945313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756594.945316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756594.945324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756594.945330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756594.945336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756594.945362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756594.945366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756594.945370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756595.137510:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756595.137517:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756595.137522:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756595.969318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756595.969373:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756595.969376:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756595.969382:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756595.969387:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756595.969392:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756595.969395:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756595.969398:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756595.969401:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756596.161542:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756596.161550:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756596.161558:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756596.993365:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756596.993376:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756596.993378:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756596.993387:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756596.993393:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756596.993398:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756596.993402:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756596.993406:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756596.993410:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756597.185393:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756597.185397:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756597.185400:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756598.017349:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756598.017360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756598.017364:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756598.017373:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756598.017379:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756598.017385:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756598.017390:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756598.017397:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756598.017403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756599.041291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756599.041305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756599.041308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756599.041316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756599.041332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756599.041337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756599.041341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756599.041346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756599.041349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756599.233495:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756599.233503:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756599.233509:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756600.065292:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756600.065336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756600.065340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756600.065364:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756600.065373:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756600.065378:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756600.065382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756600.065386:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756600.065390:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756600.257414:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756600.257423:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756600.257429:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756600.513388:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756600.513403:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756600.513407:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756600.513412:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756600.513419:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756600.513429:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9655 00000100:00080000:3.0:1787756600.513445:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756600.513450:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756600.513453:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756600.513458:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756600.513462:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756600.513466:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756600.513470:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756600.513472:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756600.513474:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756600.513481:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756600.513486:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756600.513492:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756600.513498:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756600.513504:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513509:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756600.513514:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756600.513532:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756600.513536:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513542:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756600.513548:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756600.513556:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513579:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756600.513598:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659564160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756600.513636:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756600.513645:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513675:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756600.513682:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756600.513689:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756600.513692:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756600.513696:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756600.513702:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756600.513708:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9655 00000100:00080000:3.0:1787756600.513716:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756600.513720:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756600.513723:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756600.513728:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756600.513731:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756600.513736:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756600.513740:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756600.513744:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756600.513747:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756600.513754:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756600.513759:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756600.513763:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756600.513773:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756600.513778:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513782:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756600.513791:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756600.513797:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513801:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756600.513811:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756600.513813:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513817:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756600.513820:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756600.513825:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513833:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756600.513843:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659564288/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756600.513857:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756600.513862:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513868:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756600.513873:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756600.513879:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756600.513882:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756600.513885:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756600.513891:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756600.513896:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9655 00000100:00080000:3.0:1787756600.513916:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756600.513921:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756600.513923:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756600.513927:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756600.513930:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756600.513934:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756600.513938:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756600.513942:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756600.513945:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756600.513950:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756600.513955:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756600.513958:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756600.513964:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756600.513968:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513972:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756600.513977:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756600.513982:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513985:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756600.513993:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756600.513995:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.513999:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756600.514001:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756600.514005:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.514015:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756600.514023:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659564416/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756600.514179:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756600.514236:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.514268:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756600.514275:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756600.514282:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756600.514285:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756600.514288:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756600.514294:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756600.514301:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9655 00000100:00080000:3.0:1787756600.514310:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756600.514314:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756600.514318:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756600.514323:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756600.514326:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756600.514330:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756600.514335:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756600.514339:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756600.514342:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756600.514348:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756600.514354:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756600.514357:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756600.514364:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756600.514369:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.514373:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756600.514377:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756600.514382:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.514386:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756600.514395:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756600.514398:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.514402:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756600.514405:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756600.514409:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.514419:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756600.514428:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659564544/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756600.514444:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756600.514448:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756600.514454:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756600.514929:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.514932:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756600.514934:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756600.514938:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659564416/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756600.514948:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.514951:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756600.514953:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.514955:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756600.514957:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.514959:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756600.514976:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756600.514979:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756600.514983:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756600.514985:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756600.514989:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659564416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756600.514993:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.514996:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756600.514997:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756600.515000:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756600.515003:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756600.515005:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756600.515006:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756600.515010:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756600.515012:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756600.515015:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756600.515016:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515018:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756600.515020:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756600.515022:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756600.515024:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756600.515025:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756600.515027:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756600.515029:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756600.515030:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756600.515032:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756600.515034:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515037:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515041:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756600.515043:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756600.515046:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756600.515048:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756600.515050:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756600.515052:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756600.515054:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756600.515056:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756600.515058:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756600.515061:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756600.515065:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659564544:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756600.515069:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756600.515071:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756600.515073:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756600.515075:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756600.515078:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756600.515084:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756600.515086:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787756600.515091:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756600.515104:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756600.515108:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756600.515111:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756600.515113:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756600.515115:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756600.515119:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756600.515122:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304327691 00000100:00000001:2.0:1787756600.515125:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304328385 00000100:00000001:2.0:1787756600.515128:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304327691 00000100:00000001:2.0:1787756600.515131:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304328385 00000100:00000001:2.0:1787756600.515134:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756600.515136:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756600.515138:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756600.515143:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756600/real 1787756600] req@ffff8ffb45c32a00 x1874588659564544/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756600.515154:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756600.515156:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756600.515159:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756600.515161:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756600.515162:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515164:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756600.515166:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756600.515170:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659564544/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756600.515179:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515212:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756600.515218:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515221:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756600.515223:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515227:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756600.515273:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756600.515280:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756600.515286:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756600.515292:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756600.515299:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659564544:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756600.515307:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515311:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756600.515314:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756600.515317:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756600.515322:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756600.515325:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756600.515328:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756600.515332:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756600.515334:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756600.515338:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756600.515341:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515345:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756600.515351:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756600.515354:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756600.515357:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756600.515360:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756601.089362:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756601.089376:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756601.089381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756601.089391:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756601.089399:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756601.089406:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756601.089411:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756601.089418:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756601.089423:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756601.281568:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756601.281574:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756601.281580:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756602.113300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756602.113312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756602.113317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756602.113326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756602.113333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756602.113340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756602.113344:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756602.113347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756602.113352:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756603.137293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756603.137308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756603.137331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756603.137341:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756603.137349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756603.137356:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756603.137362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756603.137370:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756603.137376:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756603.329510:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756603.329516:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756603.329522:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756604.161283:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756604.161300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756604.161304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756604.161316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756604.161324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756604.161332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756604.161336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756604.161340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756604.161344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756604.353474:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756604.353483:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756604.353490:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756605.185326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756605.185335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756605.185338:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756605.185346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756605.185351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756605.185356:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756605.185360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756605.185363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756605.185366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756605.377503:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756605.377512:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756605.377519:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756605.633774:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756605.633792:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756605.633800:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756605.633806:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756605.633813:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756605.633824:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756605.633833:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756605.633845:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756605.633866:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659564672:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756605.633884:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756605.633899:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756605.633908:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756605.633923:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756605.633946:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.633980:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756605.633992:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787756605.634013:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756605.634089:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756605.634109:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756605.634122:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756605.634131:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.634136:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756605.634146:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.634242:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304333505 00000100:00000001:2.0:1787756605.634264:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756605.634279:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304333506 00000100:00000001:2.0:1787756605.634293:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756605.634315:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756605.634322:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.634340:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756605.634360:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756605/real 1787756605] req@ffff8ffb4a536300 x1874588659564672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756605.634425:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756605.634431:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756605.634442:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756605.634458:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756605.634473:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.634492:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756605.634502:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756605.634520:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659564672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756605.634540:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.634548:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756605.634556:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.634564:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756605.634571:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.634577:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756605.634590:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756605.634606:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756605.634628:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756605.634647:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756605.634669:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659564672:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756605.634702:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.634709:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756605.634715:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756605.634722:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756605.634731:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756605.634738:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756605.634744:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756605.634752:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756605.634761:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756605.634767:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756605.634776:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.634789:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.634812:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756605.634819:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.634827:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756605.634833:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756605.634836:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756605.634841:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756605.634844:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756605.634850:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756605.634853:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756605.634857:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756605.634864:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659564800:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756605.634871:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756605.634874:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756605.634877:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756605.634879:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756605.634884:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.634894:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756605.634896:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787756605.634902:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756605.634926:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756605.634935:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756605.634941:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756605.634944:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.634947:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756605.634952:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.634960:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304332814 00000100:00000001:2.0:1787756605.634966:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304333505 00000100:00000001:2.0:1787756605.634971:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304332814 00000100:00000001:2.0:1787756605.634976:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304333506 00000100:00000001:2.0:1787756605.634980:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756605.634984:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.634988:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756605.634998:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756605/real 1787756605] req@ffff8ffb4a537800 x1874588659564800/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756605.635029:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756605.635031:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756605.635035:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756605.635038:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756605.635040:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635042:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756605.635044:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756605.635050:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659564800/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756605.635059:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635063:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756605.635065:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635067:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756605.635069:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635071:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756605.635088:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756605.635091:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756605.635095:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756605.635098:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756605.635102:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659564800:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756605.635107:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756605.635110:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756605.635112:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756605.635115:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756605.635118:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756605.635120:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756605.635124:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659564928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756605.635128:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756605.635131:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756605.635134:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756605.635136:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756605.635141:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635147:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756605.635177:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787756605.635226:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756605.635256:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756605.635266:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756605.635271:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756605.635274:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635277:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756605.635283:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635288:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304332812 00000100:00000001:2.0:1787756605.635292:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304333505 00000100:00000001:2.0:1787756605.635295:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304332812 00000100:00000001:2.0:1787756605.635298:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304333506 00000100:00000001:2.0:1787756605.635300:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756605.635303:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635305:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756605.635311:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756605/real 1787756605] req@ffff8ffb4a537b80 x1874588659564928/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756605.635321:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756605.635323:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756605.635326:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756605.635328:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756605.635330:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635332:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756605.635334:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756605.635339:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659564928/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756605.635347:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635350:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756605.635354:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635356:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756605.635358:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635360:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756605.635375:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756605.635379:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756605.635383:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756605.635386:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756605.635390:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659564928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756605.635394:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756605.635396:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756605.635398:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756605.635401:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756605.635403:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756605.635406:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756605.635410:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565056:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756605.635415:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756605.635418:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756605.635420:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756605.635422:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756605.635426:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635432:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756605.635434:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787756605.635438:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756605.635453:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756605.635458:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756605.635460:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756605.635462:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635464:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756605.635468:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635472:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304332812 00000100:00000001:2.0:1787756605.635475:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304333505 00000100:00000001:2.0:1787756605.635478:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304332812 00000100:00000001:2.0:1787756605.635481:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304333506 00000100:00000001:2.0:1787756605.635484:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756605.635486:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635488:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756605.635493:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756605/real 1787756605] req@ffff8ffb4a535c00 x1874588659565056/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756605.635502:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756605.635503:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756605.635505:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756605.635507:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756605.635509:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635511:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756605.635512:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756605.635517:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659565056/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756605.635525:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635528:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756605.635531:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635533:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756605.635534:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635536:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756605.635551:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756605.635553:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756605.635557:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756605.635560:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756605.635564:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565056:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756605.635569:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635571:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756605.635574:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756605.635576:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756605.635579:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756605.635581:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756605.635583:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756605.635587:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756605.635589:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756605.635592:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756605.635594:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635596:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756605.635598:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756605.635601:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756605.635604:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756605.635606:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756605.635608:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756605.635612:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756605.635614:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756605.635617:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756605.635620:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635623:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756605.635625:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756605.635627:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756605.635630:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756605.635632:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756605.635634:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756605.635637:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756605.635639:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756605.635643:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756605.635645:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635648:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756605.635656:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756605.635659:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756605.635663:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756605.635666:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756606.209306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756606.209315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756606.209318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756606.209324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756606.209329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756606.209334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756606.209337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756606.209340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756606.209344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756607.233344:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756607.233354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756607.233357:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756607.233363:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756607.233368:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756607.233391:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756607.233395:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756607.233400:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756607.233405:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756607.425417:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756607.425421:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756607.425425:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756608.257336:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756608.257345:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756608.257348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756608.257354:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756608.257358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756608.257363:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756608.257365:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756608.257368:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756608.257371:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756608.449456:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756608.449462:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756608.449466:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756609.281361:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756609.281374:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756609.281377:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756609.281384:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756609.281389:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756609.281396:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756609.281400:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756609.281404:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756609.281407:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756609.473513:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756609.473519:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756609.473524:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756610.305309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756610.305320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756610.305324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756610.305331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756610.305337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756610.305343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756610.305347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756610.305351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756610.305355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756610.753289:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756610.753300:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756610.753302:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756610.753306:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756610.753310:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756610.753318:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9665 00000100:00080000:3.0:1787756610.753330:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756610.753333:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756610.753335:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756610.753339:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756610.753342:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756610.753344:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756610.753346:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756610.753348:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756610.753349:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756610.753353:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756610.753355:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756610.753357:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756610.753360:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756610.753363:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753364:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756610.753366:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756610.753378:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756610.753379:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753382:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756610.753385:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756610.753390:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753404:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756610.753413:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659565184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756610.753422:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756610.753427:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753455:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756610.753457:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756610.753461:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756610.753462:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756610.753463:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756610.753465:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756610.753467:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9665 00000100:00080000:3.0:1787756610.753475:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756610.753476:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756610.753477:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756610.753479:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756610.753480:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756610.753482:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756610.753484:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756610.753485:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756610.753486:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756610.753488:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756610.753489:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756610.753493:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756610.753499:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756610.753500:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753501:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756610.753506:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756610.753509:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753511:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756610.753515:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756610.753517:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753518:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756610.753519:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756610.753521:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753524:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753530:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:3.0:1787756610.753530:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659565312/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756610.753537:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756610.753539:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756610.753541:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756610.753543:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753545:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756610.753547:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753548:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756610.753552:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787756610.753555:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787756610.753556:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:2.0:1787756610.753560:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756610.753560:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756610.753562:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787756610.753565:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753567:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756610.753572:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9665 00000100:00100000:2.0:1787756610.753576:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659565184:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787756610.753577:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756610.753592:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787756610.753592:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756610.753594:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756610.753597:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:2.0:1787756610.753598:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787756610.753598:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756610.753600:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787756610.753602:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756610.753603:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787756610.753608:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787756610.753608:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756610.753612:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787756610.753615:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756610.753615:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756610.753616:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756610.753622:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756610.753624:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756610.753626:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753628:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756610.753630:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756610.753631:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753632:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756610.753636:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787756610.753637:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787756610.753637:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753641:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:3.0:1787756610.753641:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756610.753644:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756610.753647:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753648:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787756610.753651:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756610.753657:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659565440/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756610.753661:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756610.753662:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753665:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756610.753679:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756610.753682:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756610.753683:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756610.753684:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756610.753687:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756610.753689:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9665 00000100:00080000:3.0:1787756610.753698:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756610.753700:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756610.753701:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756610.753702:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756610.753703:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756610.753705:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756610.753708:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000800:00000100:2.0:1787756610.753710:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756610.753711:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756610.753715:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756610.753718:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756610.753721:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756610.753723:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756610.753725:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756610.753727:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787756610.753728:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753731:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756610.753733:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00100000:3.0:1787756610.753733:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756610.753737:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753738:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753742:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753743:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787756610.753750:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787756610.753751:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756610.753752:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753757:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756610.753758:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756610.753759:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304338625 02000000:00000001:3.0:1787756610.753759:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753765:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756610.753769:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659565568/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756610.753774:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756610.753774:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756610.753778:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756610.753780:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753784:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304338625 00000100:00000001:2.0:1787756610.753790:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756610.753793:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756610.753795:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753798:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756610.753805:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756610/real 1787756610] req@ffff8ffb45c30700 x1874588659565184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756610.753826:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756610.753827:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756610.753830:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756610.753832:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756610.753833:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.753835:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756610.753837:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756610.753841:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659565184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756610.753851:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.753854:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756610.753856:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.753857:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756610.753858:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.753860:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756610.753861:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756610.753863:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756610.753866:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756610.753868:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756610.753870:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659565184:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756610.753874:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.753875:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756610.753876:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756610.753879:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756610.753882:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756610.753884:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756610.753897:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756610.753902:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756610.753903:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756610.753905:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756610.753906:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.753908:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.753915:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756610.753917:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753919:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756610.753920:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756610.753921:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756610.753922:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756610.753924:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756610.753925:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756610.753927:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756610.753928:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756610.753931:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565312:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756610.753933:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756610.753935:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756610.753936:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756610.753937:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756610.753939:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753943:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756610.753943:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787756610.753948:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756610.753956:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756610.753959:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756610.753960:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756610.753962:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753963:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756610.753965:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753968:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304337933 00000100:00000001:2.0:1787756610.753971:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304338625 00000100:00000001:2.0:1787756610.753973:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304337933 00000100:00000001:2.0:1787756610.753975:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304338625 00000100:00000001:2.0:1787756610.753976:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756610.753978:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.753979:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756610.753982:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756610/real 1787756610] req@ffff8ffb45c33800 x1874588659565312/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756610.753987:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756610.753988:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756610.753990:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756610.753991:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756610.753992:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.753993:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756610.753994:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756610.753996:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659565312/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756610.754006:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754007:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756610.754020:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754021:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756610.754022:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754023:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756610.754037:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756610.754039:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756610.754041:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756610.754042:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756610.754044:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565312:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756610.754047:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756610.754048:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756610.754049:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756610.754050:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756610.754052:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756610.754053:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756610.754055:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756610.754056:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756610.754057:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756610.754058:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756610.754088:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756610.754091:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754096:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756610.754097:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:2.0:1787756610.754100:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756610.754109:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756610.754112:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756610.754114:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756610.754115:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754117:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756610.754119:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754121:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304337930 00000100:00000001:2.0:1787756610.754123:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304338625 00000100:00000001:2.0:1787756610.754124:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304337930 00000100:00000001:2.0:1787756610.754126:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304338625 00000100:00000001:2.0:1787756610.754128:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756610.754129:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754130:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756610.754132:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756610/real 1787756610] req@ffff8ffb45c30e00 x1874588659565440/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756610.754137:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756610.754138:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756610.754139:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756610.754140:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756610.754141:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754142:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756610.754143:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756610.754146:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659565440/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756610.754151:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754153:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756610.754154:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754155:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756610.754156:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754157:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756610.754169:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756610.754171:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756610.754172:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756610.754174:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756610.754176:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756610.754214:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756610.754216:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756610.754218:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756610.754220:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756610.754222:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756610.754225:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756610.754229:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756610.754233:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756610.754235:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756610.754237:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756610.754239:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756610.754242:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754247:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756610.754249:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787756610.754252:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756610.754262:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756610.754265:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756610.754267:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756610.754268:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754270:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756610.754272:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754274:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304337931 00000100:00000001:2.0:1787756610.754276:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304338625 00000100:00000001:2.0:1787756610.754278:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304337931 00000100:00000001:2.0:1787756610.754280:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304338625 00000100:00000001:2.0:1787756610.754282:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756610.754283:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754284:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756610.754287:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756610/real 1787756610] req@ffff8ffb45c30000 x1874588659565568/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756610.754297:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756610.754297:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756610.754299:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756610.754300:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756610.754301:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754302:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756610.754303:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756610.754305:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659565568/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756610.754310:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754312:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756610.754313:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754314:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756610.754315:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754316:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756610.754330:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756610.754331:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756610.754333:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756610.754334:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756610.754336:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756610.754339:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754341:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756610.754342:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756610.754343:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756610.754345:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756610.754346:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756610.754347:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756610.754349:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756610.754349:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756610.754352:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756610.754352:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754354:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756610.754355:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756610.754356:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756610.754359:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756610.754360:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756610.754361:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756610.754362:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756610.754362:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756610.754363:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756610.754364:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754365:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756610.754366:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756610.754366:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756610.754367:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756610.754368:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756610.754368:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756610.754369:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756610.754370:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756610.754371:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756610.754372:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754373:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756610.754375:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756610.754376:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756610.754377:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756610.754379:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756611.329296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756611.329306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756611.329308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756611.329314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756611.329318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756611.329322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756611.329324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756611.329326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756611.329329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756611.521328:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756611.521331:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756611.521335:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756612.353313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756612.353325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756612.353328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756612.353337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756612.353342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756612.353348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756612.353352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756612.353356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756612.353360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756612.545420:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756612.545426:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756612.545443:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756613.377294:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756613.377306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756613.377309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756613.377317:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756613.377323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756613.377329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756613.377332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756613.377337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756613.377341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756613.569378:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756613.569384:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756613.569388:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756614.401268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756614.401275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756614.401277:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756614.401282:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756614.401286:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756614.401290:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756614.401292:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756614.401294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756614.401297:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.425296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787756615.425308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756615.425312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756615.425320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787756615.425325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756615.425331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756615.425335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756615.425339:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756615.425343:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756615.617422:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756615.617431:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756615.617438:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756615.873271:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756615.873279:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756615.873282:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756615.873285:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756615.873288:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756615.873294:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9670 00000100:00080000:3.0:1787756615.873303:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756615.873305:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756615.873306:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756615.873310:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756615.873312:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756615.873314:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756615.873316:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756615.873317:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756615.873318:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756615.873321:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756615.873323:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873325:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756615.873328:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756615.873330:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873331:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756615.873333:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873344:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756615.873345:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873347:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756615.873350:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756615.873354:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873365:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756615.873372:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659565696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756615.873385:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756615.873388:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873403:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756615.873405:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756615.873408:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756615.873409:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756615.873410:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756615.873412:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756615.873414:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9670 00000100:00080000:3.0:1787756615.873417:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756615.873419:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756615.873420:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756615.873422:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756615.873424:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756615.873426:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756615.873428:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756615.873430:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756615.873431:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756615.873435:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756615.873437:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873439:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756615.873445:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756615.873447:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873450:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756615.873455:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756615.873459:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873462:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873469:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756615.873471:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873474:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756615.873476:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756615.873478:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873484:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756615.873488:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659565824/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756615.873496:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756615.873498:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756615.873499:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873503:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756615.873511:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787756615.873516:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787756615.873516:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756615.873518:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756615.873525:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756615.873530:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756615.873531:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787756615.873542:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787756615.873543:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9670 00000100:00000001:2.0:1787756615.873549:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756615.873552:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756615.873559:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787756615.873561:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787756615.873561:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756615.873568:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756615.873570:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787756615.873571:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787756615.873573:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787756615.873581:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756615.873582:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756615.873597:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756615.873602:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787756615.873608:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659565696:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787756615.873612:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756615.873628:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.873630:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756615.873636:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756615.873641:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756615.873642:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756615.873649:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873650:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873662:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:2.0:1787756615.873663:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00100000:3.0:1787756615.873673:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756615.873677:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873678:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756615.873682:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873683:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756615.873687:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873690:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756615.873692:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756615.873694:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873701:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756615.873707:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659565952/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756615.873713:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756615.873715:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756615.873716:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.873724:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:3.0:1787756615.873727:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.873732:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787756615.873733:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756615.873741:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756615.873743:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756615.873744:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756615.873748:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756615.873750:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9670 00000100:00080000:3.0:1787756615.873782:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756615.873783:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756615.873784:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756615.873786:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756615.873787:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756615.873789:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756615.873791:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756615.873793:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756615.873798:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:2.0:1787756615.873799:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787756615.873809:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756615.873813:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873815:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756615.873819:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756615.873821:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873823:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756615.873825:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00100000:3.0:1787756615.873826:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756615.873833:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.873836:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756615.873836:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.873841:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873843:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787756615.873848:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787756615.873850:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873855:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756615.873858:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756615.873860:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756615.873861:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.873880:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304343745 00000100:00000001:3.0:1787756615.873881:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756615.873899:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659566080/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756615.873905:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756615.873906:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756615.873912:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756615.873919:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.873923:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304343745 00000100:00000001:2.0:1787756615.873931:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756615.873937:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756615.873941:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.873945:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756615.873955:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756615/real 1787756615] req@ffff8ffb45c33480 x1874588659565696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756615.874012:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756615.874015:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756615.874021:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756615.874025:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756615.874027:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874031:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756615.874034:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756615.874042:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659565696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756615.874053:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874058:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756615.874062:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874064:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756615.874066:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874069:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756615.874072:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756615.874077:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756615.874083:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756615.874087:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756615.874092:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659565696:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756615.874098:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756615.874101:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756615.874102:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756615.874105:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756615.874108:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756615.874111:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756615.874114:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565824:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756615.874119:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756615.874122:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756615.874124:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756615.874127:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756615.874131:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874138:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756615.874140:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:2.0:1787756615.874147:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756615.874165:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756615.874172:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756615.874174:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756615.874176:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874179:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756615.874228:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874236:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304343054 00000100:00000001:2.0:1787756615.874240:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304343745 00000100:00000001:2.0:1787756615.874243:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304343054 00000100:00000001:2.0:1787756615.874246:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304343745 00000100:00000001:2.0:1787756615.874249:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756615.874252:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874255:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756615.874262:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756615/real 1787756615] req@ffff8ffb45c31500 x1874588659565824/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756615.874273:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756615.874275:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756615.874278:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756615.874281:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756615.874283:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874285:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756615.874287:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756615.874292:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659565824/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756615.874301:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874303:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756615.874306:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874308:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756615.874310:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874312:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756615.874333:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756615.874336:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756615.874341:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756615.874344:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756615.874348:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565824:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756615.874353:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874356:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756615.874358:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756615.874361:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756615.874367:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756615.874371:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756615.874374:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756615.874380:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756615.874382:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756615.874387:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756615.874389:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874391:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756615.874392:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756615.874394:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756615.874396:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756615.874398:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756615.874399:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756615.874401:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756615.874402:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756615.874404:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756615.874406:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874409:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874420:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756615.874424:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874429:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756615.874431:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756615.874434:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756615.874437:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756615.874439:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756615.874443:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756615.874446:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756615.874449:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756615.874456:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565952:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756615.874463:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756615.874467:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756615.874470:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756615.874474:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756615.874480:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874491:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756615.874494:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:2.0:1787756615.874501:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756615.874526:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756615.874535:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756615.874539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756615.874541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874543:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756615.874547:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874552:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304343051 00000100:00000001:2.0:1787756615.874556:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304343745 00000100:00000001:2.0:1787756615.874560:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304343051 00000100:00000001:2.0:1787756615.874563:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304343745 00000100:00000001:2.0:1787756615.874565:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756615.874568:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874570:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756615.874576:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756615/real 1787756615] req@ffff8ffb45c31180 x1874588659565952/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756615.874586:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756615.874587:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756615.874591:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756615.874593:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756615.874595:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874597:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756615.874599:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756615.874603:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659565952/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756615.874613:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874616:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756615.874618:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874620:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756615.874622:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874624:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756615.874641:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756615.874645:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756615.874648:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756615.874651:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756615.874654:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659565952:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756615.874659:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756615.874662:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756615.874663:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756615.874667:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756615.874669:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756615.874672:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756615.874675:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756615.874679:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756615.874681:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756615.874683:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756615.874685:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756615.874689:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874695:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756615.874696:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787756615.874702:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756615.874716:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756615.874721:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756615.874724:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756615.874726:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874728:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756615.874731:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874735:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304343051 00000100:00000001:2.0:1787756615.874738:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304343745 00000100:00000001:2.0:1787756615.874741:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304343051 00000100:00000001:2.0:1787756615.874744:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304343745 00000100:00000001:2.0:1787756615.874746:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756615.874749:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874751:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756615.874755:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756615/real 1787756615] req@ffff8ffb45c30000 x1874588659566080/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756615.874765:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756615.874766:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756615.874769:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756615.874771:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756615.874774:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874776:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756615.874778:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756615.874782:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659566080/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756615.874791:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874793:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756615.874796:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874798:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756615.874799:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874801:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756615.874819:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756615.874822:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756615.874825:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756615.874828:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756615.874831:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756615.874835:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874838:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756615.874840:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756615.874842:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756615.874844:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756615.874847:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756615.874848:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756615.874851:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756615.874853:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756615.874855:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756615.874857:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874859:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756615.874860:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756615.874862:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756615.874864:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756615.874865:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756615.874867:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756615.874869:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756615.874870:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756615.874873:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756615.874875:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874877:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756615.874882:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756615.874884:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756615.874887:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756615.874889:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756616.641464:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756616.641475:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756616.641484:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756617.665488:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756617.665495:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756617.665501:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756619.713481:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756619.713487:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756619.713493:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756620.737572:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756620.737581:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756620.737587:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756620.993574:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756620.993582:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756620.993587:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756620.993592:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756620.993596:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756620.993604:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756620.993609:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756620.993616:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756620.993627:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659566208:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756620.993638:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756620.993644:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756620.993650:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756620.993656:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756620.993684:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.993708:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756620.993715:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133081856 : -123165576469760 : ffff8ffb4790e300) 00000100:00000001:2.0:1787756620.993725:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756620.993774:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756620.993790:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756620.993799:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756620.993806:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.993814:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756620.993822:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.993830:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304348865 00000100:00000001:2.0:1787756620.993860:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756620.993874:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304348865 00000100:00000001:2.0:1787756620.993909:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756620.993929:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756620.993936:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.993946:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756620.993962:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756620/real 1787756620] req@ffff8ffb4790e300 x1874588659566208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756620.993991:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756620.993996:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756620.994004:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756620.994009:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756620.994012:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994015:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756620.994017:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756620.994023:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e300 x1874588659566208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756620.994032:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994035:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756620.994038:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994040:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756620.994041:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994044:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756620.994046:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756620.994049:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756620.994052:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756620.994054:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756620.994058:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659566208:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756620.994062:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994065:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756620.994066:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756620.994069:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756620.994073:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756620.994075:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756620.994077:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756620.994082:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756620.994083:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756620.994086:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756620.994088:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994090:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994096:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756620.994098:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994100:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756620.994102:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756620.994103:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756620.994105:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756620.994106:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756620.994108:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756620.994110:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756620.994113:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756620.994115:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566336:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756620.994118:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756620.994120:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756620.994122:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756620.994123:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756620.994143:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994150:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756620.994152:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133087232 : -123165576464384 : ffff8ffb4790f800) 00000100:00000001:2.0:1787756620.994157:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756620.994172:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756620.994178:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756620.994181:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756620.994182:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994184:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756620.994187:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994192:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304348173 00000100:00000001:2.0:1787756620.994230:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304348865 00000100:00000001:2.0:1787756620.994234:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304348174 00000100:00000001:2.0:1787756620.994237:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304348865 00000100:00000001:2.0:1787756620.994239:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756620.994241:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994243:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756620.994249:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756620/real 1787756620] req@ffff8ffb4790f800 x1874588659566336/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756620.994260:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756620.994261:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756620.994265:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756620.994267:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756620.994268:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994270:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756620.994272:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756620.994276:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f800 x1874588659566336/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756620.994296:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994299:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756620.994301:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994303:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756620.994305:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994307:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756620.994321:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756620.994325:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756620.994328:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756620.994331:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756620.994334:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566336:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756620.994339:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756620.994340:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756620.994342:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756620.994345:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756620.994347:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756620.994349:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756620.994352:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566464:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756620.994356:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756620.994358:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756620.994361:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756620.994363:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756620.994367:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994372:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756620.994374:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133078272 : -123165576473344 : ffff8ffb4790d500) 00000100:00000001:2.0:1787756620.994378:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756620.994390:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756620.994395:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756620.994397:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756620.994399:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994400:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756620.994403:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994407:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304348171 00000100:00000001:2.0:1787756620.994410:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304348865 00000100:00000001:2.0:1787756620.994412:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304348171 00000100:00000001:2.0:1787756620.994415:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304348865 00000100:00000001:2.0:1787756620.994417:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756620.994419:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994421:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756620.994425:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756620/real 1787756620] req@ffff8ffb4790d500 x1874588659566464/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756620.994433:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756620.994434:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756620.994436:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756620.994438:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756620.994439:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994441:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756620.994443:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756620.994447:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d500 x1874588659566464/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756620.994454:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994457:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756620.994460:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994462:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756620.994463:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994465:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756620.994477:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756620.994480:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756620.994484:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756620.994486:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756620.994489:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566464:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756620.994493:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756620.994494:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756620.994496:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756620.994498:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756620.994499:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756620.994501:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756620.994504:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756620.994507:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756620.994508:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756620.994510:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756620.994512:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756620.994515:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994519:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756620.994521:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:2.0:1787756620.994524:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756620.994534:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756620.994537:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756620.994539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756620.994541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994543:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756620.994546:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994566:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304348171 00000100:00000001:2.0:1787756620.994568:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304348865 00000100:00000001:2.0:1787756620.994570:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304348171 00000100:00000001:2.0:1787756620.994573:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304348865 00000100:00000001:2.0:1787756620.994576:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756620.994577:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994579:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756620.994583:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756620/real 1787756620] req@ffff8ffb4790ce00 x1874588659566592/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756620.994589:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756620.994590:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756620.994592:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756620.994593:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756620.994594:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994596:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756620.994597:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756620.994600:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659566592/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756620.994606:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994608:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756620.994611:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994612:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756620.994613:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994615:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756620.994626:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756620.994628:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756620.994631:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756620.994633:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756620.994636:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756620.994639:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994641:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756620.994643:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756620.994644:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756620.994646:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756620.994664:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756620.994666:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756620.994668:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756620.994670:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756620.994672:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756620.994673:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994675:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756620.994677:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756620.994678:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756620.994680:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756620.994681:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756620.994683:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756620.994684:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756620.994685:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756620.994687:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756620.994688:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994690:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756620.994691:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756620.994692:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756620.994694:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756620.994695:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756620.994696:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756620.994697:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756620.994698:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756620.994700:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756620.994702:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994704:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756620.994707:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756620.994709:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756620.994711:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756620.994712:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756621.761453:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756621.761462:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756621.761469:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756623.809439:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756623.809446:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756623.809452:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756624.833469:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756624.833475:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756624.833480:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756625.857513:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756625.857522:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756625.857529:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756626.113747:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756626.113760:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756626.113768:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756626.113781:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756626.113806:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756626.113817:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756626.113835:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756626.113840:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756626.113856:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659566720:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756626.113873:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756626.113886:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756626.113892:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756626.113901:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756626.113914:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.113958:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756626.113964:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:2.0:1787756626.113974:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756626.114043:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756626.114076:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756626.114088:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756626.114099:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.114108:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756626.114124:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.114145:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304353985 00000100:00000001:2.0:1787756626.114169:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756626.114263:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304353986 00000100:00000001:2.0:1787756626.114295:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756626.114311:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756626.114323:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.114335:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756626.114371:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756626/real 1787756626] req@ffff8ffb4790e680 x1874588659566720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756626.114429:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756626.114437:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756626.114454:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756626.114466:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756626.114478:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114488:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756626.114494:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756626.114512:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659566720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756626.114579:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114589:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756626.114602:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114605:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756626.114608:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114613:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756626.114617:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756626.114621:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756626.114629:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756626.114633:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756626.114640:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659566720:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756626.114649:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114653:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756626.114656:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756626.114660:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756626.114666:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756626.114670:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756626.114673:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756626.114680:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756626.114682:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756626.114686:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756626.114688:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114691:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114701:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756626.114704:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.114707:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756626.114709:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756626.114712:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756626.114713:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756626.114716:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756626.114719:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756626.114722:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756626.114724:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756626.114729:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566848:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756626.114734:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756626.114737:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756626.114739:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756626.114742:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756626.114746:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.114755:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756626.114757:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:2.0:1787756626.114764:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756626.114785:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756626.114815:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756626.114819:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756626.114821:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.114824:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756626.114828:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.114834:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304353294 00000100:00000001:2.0:1787756626.114839:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304353985 00000100:00000001:2.0:1787756626.114842:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304353294 00000100:00000001:2.0:1787756626.114845:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304353986 00000100:00000001:2.0:1787756626.114848:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756626.114850:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.114853:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756626.114859:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756626/real 1787756626] req@ffff8ffb4790d880 x1874588659566848/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756626.114870:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756626.114872:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756626.114875:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756626.114877:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756626.114879:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114881:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756626.114883:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756626.114888:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659566848/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756626.114917:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114920:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756626.114923:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114925:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756626.114927:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.114929:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756626.114951:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756626.114954:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756626.114957:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756626.114960:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756626.114964:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566848:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756626.114969:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756626.114971:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756626.114973:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756626.114976:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756626.114978:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756626.114981:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756626.114984:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566976:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756626.114988:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756626.114991:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756626.114993:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756626.114995:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756626.114999:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115005:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756626.115007:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:2.0:1787756626.115012:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756626.115026:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756626.115032:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756626.115035:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756626.115036:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115039:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756626.115042:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115046:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304353291 00000100:00000001:2.0:1787756626.115050:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304353985 00000100:00000001:2.0:1787756626.115063:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304353291 00000100:00000001:2.0:1787756626.115066:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304353986 00000100:00000001:2.0:1787756626.115069:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756626.115071:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115073:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756626.115078:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756626/real 1787756626] req@ffff8ffb4790c380 x1874588659566976/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756626.115087:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756626.115088:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756626.115091:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756626.115093:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756626.115095:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115097:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756626.115099:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756626.115103:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659566976/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756626.115111:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115114:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756626.115117:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115119:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756626.115120:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115122:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756626.115146:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756626.115149:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756626.115152:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756626.115154:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756626.115159:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659566976:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756626.115164:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756626.115166:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756626.115167:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756626.115170:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756626.115172:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756626.115174:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756626.115178:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659567104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756626.115181:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756626.115183:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756626.115215:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756626.115220:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756626.115227:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115235:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756626.115238:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:2.0:1787756626.115244:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756626.115264:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756626.115273:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756626.115276:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756626.115289:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115292:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756626.115295:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115300:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304353292 00000100:00000001:2.0:1787756626.115304:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304353985 00000100:00000001:2.0:1787756626.115308:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304353292 00000100:00000001:2.0:1787756626.115312:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304353986 00000100:00000001:2.0:1787756626.115315:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756626.115318:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115320:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756626.115325:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756626/real 1787756626] req@ffff8ffb4790f100 x1874588659567104/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756626.115343:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756626.115345:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756626.115348:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756626.115351:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756626.115352:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115354:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756626.115356:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756626.115361:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659567104/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756626.115370:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115373:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756626.115377:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115379:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756626.115381:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115383:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756626.115401:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756626.115404:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756626.115407:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756626.115410:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756626.115413:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659567104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756626.115417:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115419:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756626.115421:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756626.115423:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756626.115426:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756626.115428:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756626.115430:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756626.115433:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756626.115435:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756626.115438:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756626.115440:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115443:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756626.115445:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756626.115446:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756626.115450:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756626.115452:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756626.115453:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756626.115455:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756626.115457:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756626.115459:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756626.115460:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115462:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756626.115463:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756626.115465:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756626.115466:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756626.115468:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756626.115469:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756626.115471:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756626.115472:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756626.115474:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756626.115476:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115478:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756626.115482:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756626.115484:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756626.115486:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756626.115488:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756628.929483:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756628.929495:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756628.929505:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756629.953513:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756629.953522:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756629.953529:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756631.233687:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756631.233726:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756631.233736:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756631.233745:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756631.233761:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756631.233786:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756631.233800:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756631.233814:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756631.233838:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659567232:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756631.233859:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756631.233877:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756631.233883:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756631.233890:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756631.233911:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.233947:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756631.233955:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:2.0:1787756631.233973:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756631.234055:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756631.234079:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756631.234091:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756631.234100:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234106:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756631.234122:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234131:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304359105 00000100:00000001:2.0:1787756631.234165:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756631.234257:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304359106 00000100:00000001:2.0:1787756631.234272:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756631.234298:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756631.234316:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234331:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756631.234348:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756631/real 1787756631] req@ffff8ffb4790d180 x1874588659567232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756631.234417:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756631.234424:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756631.234434:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756631.234439:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756631.234443:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234448:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756631.234451:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756631.234461:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659567232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756631.234474:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234479:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756631.234483:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234485:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756631.234487:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234490:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756631.234493:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756631.234499:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756631.234505:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756631.234509:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756631.234515:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659567232:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756631.234521:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756631.234523:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756631.234525:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756631.234528:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756631.234531:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756631.234534:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756631.234538:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659567360:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756631.234543:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756631.234546:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756631.234548:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756631.234550:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756631.234555:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234562:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756631.234564:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:2.0:1787756631.234571:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756631.234588:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756631.234594:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756631.234597:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756631.234599:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234602:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756631.234606:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234611:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304358414 00000100:00000001:2.0:1787756631.234615:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304359105 00000100:00000001:2.0:1787756631.234618:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304358414 00000100:00000001:2.0:1787756631.234622:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304359106 00000100:00000001:2.0:1787756631.234625:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756631.234627:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234631:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756631.234636:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756631/real 1787756631] req@ffff8ffb4790fb80 x1874588659567360/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756631.234647:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756631.234649:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756631.234652:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756631.234654:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756631.234656:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234658:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756631.234660:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756631.234665:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659567360/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756631.234673:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234675:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756631.234678:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234680:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756631.234682:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234684:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756631.234729:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756631.234734:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756631.234738:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756631.234742:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756631.234746:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659567360:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756631.234751:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234754:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756631.234756:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756631.234761:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756631.234767:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756631.234771:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756631.234775:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756631.234782:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756631.234785:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756631.234790:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756631.234792:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234794:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756631.234795:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756631.234797:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756631.234799:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756631.234801:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756631.234802:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756631.234804:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756631.234805:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756631.234808:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756631.234809:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234812:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234822:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756631.234825:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234828:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756631.234830:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756631.234832:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756631.234834:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756631.234836:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756631.234839:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756631.234841:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756631.234844:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756631.234848:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659567488:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756631.234852:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756631.234854:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756631.234857:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756631.234859:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756631.234863:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234870:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756631.234872:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:2.0:1787756631.234878:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756631.234892:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756631.234897:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756631.234900:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756631.234902:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234904:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756631.234908:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234912:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304358411 00000100:00000001:2.0:1787756631.234916:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304359105 00000100:00000001:2.0:1787756631.234920:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304358411 00000100:00000001:2.0:1787756631.234922:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304359106 00000100:00000001:2.0:1787756631.234925:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756631.234928:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.234930:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756631.234935:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756631/real 1787756631] req@ffff8ffb4790ed80 x1874588659567488/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756631.234944:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756631.234945:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756631.234948:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756631.234950:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756631.234952:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234955:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756631.234957:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756631.234961:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659567488/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756631.234970:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234972:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756631.234975:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234977:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756631.234978:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.234980:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756631.235000:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756631.235003:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756631.235008:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756631.235010:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756631.235015:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659567488:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756631.235019:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756631.235022:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756631.235024:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756631.235027:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756631.235029:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756631.235032:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756631.235035:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659567616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756631.235039:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756631.235041:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756631.235043:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756631.235045:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756631.235047:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.235052:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756631.235054:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:2.0:1787756631.235059:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756631.235070:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756631.235074:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756631.235077:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756631.235079:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.235081:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756631.235084:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.235088:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304358411 00000100:00000001:2.0:1787756631.235091:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304359105 00000100:00000001:2.0:1787756631.235094:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304358411 00000100:00000001:2.0:1787756631.235097:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304359106 00000100:00000001:2.0:1787756631.235100:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756631.235102:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.235104:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756631.235108:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756631/real 1787756631] req@ffff8ffb4790ca80 x1874588659567616/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756631.235117:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756631.235118:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756631.235121:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756631.235123:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756631.235126:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.235128:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756631.235130:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756631.235134:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659567616/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756631.235143:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.235145:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756631.235148:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.235150:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756631.235151:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.235153:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756631.235172:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756631.235175:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756631.235178:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756631.235181:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756631.235216:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659567616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756631.235222:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.235224:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756631.235226:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756631.235228:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756631.235231:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756631.235233:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756631.235235:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756631.235238:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756631.235240:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756631.235243:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756631.235245:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.235247:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756631.235248:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756631.235250:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756631.235252:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756631.235254:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756631.235255:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756631.235258:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756631.235259:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756631.235262:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756631.235264:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.235266:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756631.235270:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756631.235272:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756631.235275:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756631.235277:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756633.025446:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756633.025455:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756633.025462:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756634.049413:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756634.049425:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756634.049434:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756636.353377:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756636.353386:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756636.353388:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756636.353391:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756636.353394:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756636.353400:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9691 00000100:00080000:2.0:1787756636.353408:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756636.353411:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756636.353412:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756636.353416:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756636.353417:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756636.353420:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756636.353421:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756636.353423:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756636.353425:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756636.353428:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756636.353429:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756636.353431:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756636.353434:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756636.353436:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353437:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756636.353439:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756636.353449:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756636.353450:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353452:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756636.353455:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756636.353459:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353469:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756636.353476:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659567744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756636.353484:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756636.353487:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353502:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756636.353504:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756636.353507:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756636.353508:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756636.353509:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756636.353511:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756636.353513:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9691 00000100:00080000:2.0:1787756636.353515:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756636.353516:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756636.353518:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756636.353520:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756636.353521:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756636.353535:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756636.353537:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756636.353538:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756636.353539:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756636.353542:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756636.353543:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756636.353544:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756636.353548:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756636.353552:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353554:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756636.353558:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756636.353561:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353564:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756636.353570:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756636.353572:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353575:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756636.353576:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756636.353578:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353585:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756636.353594:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659567872/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756636.353622:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756636.353625:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353641:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756636.353647:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756636.353650:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756636.353654:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756636.353655:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756636.353658:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756636.353662:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9691 00000100:00080000:2.0:1787756636.353666:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756636.353674:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756636.353679:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756636.353683:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756636.353685:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756636.353688:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756636.353691:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756636.353705:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756636.353706:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756636.353710:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756636.353714:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756636.353716:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756636.353720:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756636.353726:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353731:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756636.353735:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756636.353738:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353742:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756636.353749:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756636.353751:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353754:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756636.353757:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756636.353760:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353769:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756636.353775:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659568000/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756636.353786:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756636.353788:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353792:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756636.353794:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756636.353796:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756636.353797:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756636.353799:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756636.353805:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756636.353810:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9691 00000100:00080000:2.0:1787756636.353815:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756636.353818:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756636.353821:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756636.353827:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756636.353830:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756636.353833:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756636.353837:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756636.353841:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756636.353845:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756636.353852:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756636.353857:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756636.353860:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756636.353864:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756636.353867:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353877:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756636.353879:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756636.353903:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353910:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756636.353919:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756636.353925:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353932:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756636.353939:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756636.353945:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.353975:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756636.353996:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659568128/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756636.354030:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756636.354038:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756636.354047:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756637.121431:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756637.121439:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756637.121445:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756638.145385:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756638.145398:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756638.145402:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756641.217488:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756641.217495:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756641.217500:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756641.473419:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756641.473435:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756641.473439:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756641.473445:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756641.473452:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756641.473464:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9696 00000100:00080000:3.0:1787756641.473481:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756641.473485:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756641.473489:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756641.473495:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756641.473499:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756641.473504:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756641.473508:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756641.473510:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756641.473512:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756641.473540:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756641.473544:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756641.473548:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756641.473553:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756641.473558:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473561:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756641.473565:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756641.473585:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756641.473588:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473593:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756641.473598:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756641.473605:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473625:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756641.473640:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659568256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756641.473655:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756641.473661:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473684:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756641.473689:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756641.473694:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756641.473696:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756641.473699:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756641.473702:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756641.473705:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9696 00000100:00080000:3.0:1787756641.473709:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756641.473713:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756641.473715:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756641.473718:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756641.473720:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756641.473723:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756641.473726:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756641.473728:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756641.473730:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756641.473734:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756641.473737:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756641.473740:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756641.473748:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756641.473751:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473755:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756641.473763:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756641.473767:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473771:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756641.473777:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756641.473782:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473785:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756641.473793:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756641.473802:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473816:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756641.473830:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659568384/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756641.473852:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756641.473860:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473871:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756641.473878:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756641.473889:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756641.473896:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756641.473901:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756641.473911:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756641.473922:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9696 00000100:00080000:3.0:1787756641.473927:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756641.473930:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756641.473932:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756641.473935:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756641.473937:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756641.473939:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756641.473942:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756641.473945:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756641.473952:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756641.473963:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756641.473969:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756641.473974:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756641.473978:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756641.473981:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473984:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756641.473987:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756641.473991:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.473994:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756641.474000:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756641.474002:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.474004:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756641.474005:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756641.474009:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.474016:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756641.474022:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659568512/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756641.474033:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756641.474041:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.474046:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756641.474053:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756641.474059:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756641.474062:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756641.474065:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756641.474070:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756641.474082:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9696 00000100:00080000:3.0:1787756641.474094:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756641.474103:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756641.474106:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756641.474109:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756641.474112:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756641.474118:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756641.474123:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756641.474131:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756641.474133:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756641.474138:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756641.474143:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756641.474164:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756641.474171:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756641.474178:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.474236:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756641.474239:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756641.474245:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.474251:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756641.474263:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756641.474270:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.474275:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756641.474280:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756641.474284:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.474295:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756641.474314:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659568640/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756641.474352:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756641.474358:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756641.474362:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756642.241584:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756642.241590:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756642.241594:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756645.313438:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756645.313445:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756645.313471:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756646.337532:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756646.337541:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756646.337548:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756646.593335:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756646.593349:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756646.593353:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756646.593358:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756646.593364:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756646.593373:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9701 00000100:00080000:3.0:1787756646.593387:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756646.593391:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756646.593394:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756646.593399:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756646.593402:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756646.593406:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756646.593409:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756646.593412:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756646.593414:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756646.593420:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756646.593424:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756646.593468:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756646.593474:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756646.593479:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593482:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756646.593485:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756646.593500:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756646.593502:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593506:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756646.593510:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756646.593515:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593530:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756646.593543:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659568768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756646.593557:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756646.593563:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593584:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756646.593589:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756646.593594:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756646.593596:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756646.593598:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756646.593602:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756646.593605:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9701 00000100:00080000:3.0:1787756646.593610:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756646.593612:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756646.593614:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756646.593618:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756646.593619:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756646.593623:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756646.593625:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756646.593628:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756646.593630:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756646.593635:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756646.593638:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756646.593642:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756646.593649:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756646.593652:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593655:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756646.593662:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756646.593666:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593670:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756646.593682:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756646.593689:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593692:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756646.593697:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756646.593705:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593734:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756646.593749:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659568896/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756646.593777:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756646.593785:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593796:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756646.593804:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756646.593817:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756646.593823:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756646.593825:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756646.593829:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756646.593840:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9701 00000100:00080000:3.0:1787756646.593846:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756646.593848:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756646.593850:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756646.593853:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756646.593856:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756646.593865:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756646.593876:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756646.593881:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756646.593886:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756646.593891:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756646.593894:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756646.593897:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756646.593901:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756646.593904:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593906:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756646.593909:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756646.593913:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593916:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756646.593923:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756646.593925:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593927:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756646.593932:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756646.593938:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593950:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756646.593960:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659569024/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756646.593984:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756646.593991:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.593998:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756646.594010:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756646.594019:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756646.594023:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756646.594025:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756646.594028:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756646.594033:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9701 00000100:00080000:3.0:1787756646.594056:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756646.594066:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756646.594071:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756646.594081:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756646.594095:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756646.594103:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756646.594113:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756646.594123:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756646.594128:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756646.594142:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756646.594148:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756646.594158:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756646.594164:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756646.594173:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.594179:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756646.594234:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756646.594245:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.594252:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756646.594267:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756646.594273:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.594282:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756646.594290:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756646.594298:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.594314:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756646.594329:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659569152/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756646.594359:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756646.594367:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756646.594377:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756649.409470:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756649.409478:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756649.409485:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756650.433524:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756650.433530:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756650.433534:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756651.713277:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756651.713290:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756651.713294:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756651.713299:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756651.713304:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756651.713327:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9706 00000100:00080000:3.0:1787756651.713363:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756651.713368:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756651.713371:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756651.713376:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756651.713379:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756651.713383:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756651.713386:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756651.713389:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756651.713391:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756651.713397:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756651.713400:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756651.713404:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756651.713408:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756651.713412:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713415:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756651.713418:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756651.713431:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756651.713434:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713438:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756651.713441:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756651.713446:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713460:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756651.713473:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659569280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756651.713487:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756651.713493:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713513:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756651.713517:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756651.713525:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756651.713527:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756651.713529:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756651.713533:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756651.713536:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9706 00000100:00080000:3.0:1787756651.713541:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756651.713544:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756651.713546:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756651.713550:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756651.713552:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756651.713556:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756651.713559:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756651.713561:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756651.713564:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756651.713568:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756651.713572:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756651.713575:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756651.713585:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756651.713591:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713596:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756651.713602:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756651.713609:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713615:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756651.713621:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756651.713626:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713630:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756651.713633:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756651.713640:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713648:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756651.713661:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659569408/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756651.713686:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756651.713689:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713694:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756651.713701:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756651.713712:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756651.713717:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756651.713723:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756651.713727:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756651.713733:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9706 00000100:00080000:3.0:1787756651.713741:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756651.713747:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756651.713752:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756651.713759:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756651.713761:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756651.713764:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756651.713768:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756651.713772:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756651.713778:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756651.713784:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756651.713786:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756651.713788:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756651.713792:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756651.713794:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713796:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756651.713799:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756651.713801:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713804:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756651.713811:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756651.713813:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713816:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756651.713818:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756651.713821:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713832:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756651.713842:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659569536/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756651.713855:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756651.713858:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713861:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756651.713865:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756651.713869:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756651.713870:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756651.713872:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756651.713876:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756651.713879:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9706 00000100:00080000:3.0:1787756651.713884:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756651.713887:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756651.713888:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756651.713891:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756651.713893:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756651.713897:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756651.713903:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756651.713911:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756651.713913:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756651.713920:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756651.713924:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756651.713930:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756651.713935:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756651.713941:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713943:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756651.713945:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756651.713948:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713950:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756651.713956:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756651.713960:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713963:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756651.713965:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756651.713967:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.713976:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756651.713987:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659569664/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756651.713998:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756651.714005:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756651.714010:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756653.505522:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756653.505529:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756653.505534:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756654.529483:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756654.529491:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756654.529498:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756657.601596:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756657.601605:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756657.601611:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756658.625511:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756658.625520:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756658.625527:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756661.697464:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756661.697474:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756661.697482:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756661.953679:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756661.953691:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756661.953700:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756661.953708:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756661.953717:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756661.953731:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756661.953739:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756661.953749:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756661.953762:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659570304:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756661.953779:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756661.953787:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756661.953793:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756661.953814:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756661.953825:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.953854:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756661.953861:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787756661.953873:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756661.953958:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.953972:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304389825 00000100:00000001:2.0:1787756661.953992:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756661.954010:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304389825 00000100:00000001:2.0:1787756661.954022:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756661.954033:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756661.954040:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954049:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954056:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756661.954072:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756661.954078:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787756661.954092:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756661.954097:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756661.954103:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756661.954109:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756661.954117:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756661.954125:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756661.954131:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756661.954137:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756661.954145:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659570432:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756661.954154:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756661.954225:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756661.954228:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756661.954238:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756661.954249:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954266:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756661.954271:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787756661.954280:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756661.954305:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954316:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304389134 00000100:00000001:2.0:1787756661.954325:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304389825 00000100:00000001:2.0:1787756661.954333:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304389134 00000100:00000001:2.0:1787756661.954341:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304389825 00000100:00000001:2.0:1787756661.954352:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756661.954356:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954363:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954372:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756661.954389:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756661.954399:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787756661.954414:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756661.954423:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756661.954464:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756661.954473:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756661.954480:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756661.954492:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756661.954499:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756661.954505:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756661.954513:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659570560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756661.954521:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756661.954528:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756661.954536:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756661.954545:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756661.954555:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954573:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756661.954579:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787756661.954586:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756661.954609:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954616:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304389131 00000100:00000001:2.0:1787756661.954621:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304389825 00000100:00000001:2.0:1787756661.954625:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304389131 00000100:00000001:2.0:1787756661.954630:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304389825 00000100:00000001:2.0:1787756661.954636:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756661.954642:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954651:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954656:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756661.954663:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756661.954666:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787756661.954671:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756661.954674:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756661.954677:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756661.954681:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756661.954685:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756661.954690:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756661.954694:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756661.954698:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756661.954707:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659570688:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756661.954714:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756661.954717:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756661.954721:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756661.954724:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756661.954731:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954743:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756661.954746:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787756661.954753:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756661.954777:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954784:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304389131 00000100:00000001:2.0:1787756661.954789:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304389825 00000100:00000001:2.0:1787756661.954793:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304389131 00000100:00000001:2.0:1787756661.954798:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304389825 00000100:00000001:2.0:1787756661.954803:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756661.954807:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954812:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954816:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756661.954823:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756661.954826:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787756661.954830:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756661.954833:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756661.954837:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756661.954841:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756662.721498:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756662.721510:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756662.721520:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787756665.025735:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787756665.025816:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787756665.025864:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756665.025893:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756665.025933:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756665.025939:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756665.025944:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756665.025964:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756665.025976:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756665.025985:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756665.025991:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756665.025995:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756665.026010:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787756665.026017:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756665.026021:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756665.026035:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756665.026038:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756665.026045:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756665.026048:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787756665.026051:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756665.026066:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756665.026075:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756665.026081:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756665.026084:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756665.026087:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:2.0:1787756665.026140:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756661/real 1787756665] req@ffff8ffb4a537b80 x1874588659570560/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756665.026176:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756665.026181:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756665.026240:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756665.026250:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756665.026257:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026264:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756665.026269:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756665.026282:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659570560/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756665.026300:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026306:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756665.026314:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026319:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756665.026322:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026326:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756665.026351:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756665.026358:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756665.026368:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756665.026374:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756665.026382:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659570560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756665.026393:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756665.026407:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756661/real 1787756665] req@ffff8ffb4a536300 x1874588659570432/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756665.026427:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756665.026431:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756665.026439:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756665.026445:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756665.026449:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026456:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756665.026459:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756665.026468:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659570432/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756665.026484:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026489:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756665.026497:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026501:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756665.026504:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026508:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756665.026530:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756665.026536:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756665.026544:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756665.026549:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756665.026556:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659570432:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756665.026566:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756665.026574:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756661/real 1787756665] req@ffff8ffb4a537800 x1874588659570304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756665.026591:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756665.026594:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756665.026601:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756665.026604:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756665.026607:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026611:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756665.026613:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756665.026622:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659570304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756665.026636:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026640:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756665.026645:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026649:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756665.026652:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026656:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756665.026676:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756665.026681:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756665.026689:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756665.026693:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756665.026700:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659570304:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756665.026709:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756665.026714:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756665.026718:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756665.026724:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756665.026738:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756665.026743:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756665.026747:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756665.026754:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756665.026758:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756665.026764:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756665.026767:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026771:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756665.026774:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756665.026776:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756665.026781:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756665.026783:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756665.026786:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756665.026791:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756665.026793:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756665.026796:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756665.026799:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026803:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756665.026805:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756665.026807:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756665.026810:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756665.026812:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756665.026814:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756665.026818:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756665.026820:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756665.026823:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756665.026826:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026832:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756665.026836:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756665.026839:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756665.026842:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756665.026853:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756661/real 1787756665] req@ffff8ffb4a535c00 x1874588659570688/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756665.026873:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756665.026875:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756665.026882:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756665.026885:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756665.026888:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026891:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756665.026894:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756665.026904:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659570688/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756665.026914:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026917:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756665.026921:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026922:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756665.026924:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026926:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756665.026940:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756665.026943:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756665.026948:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756665.026950:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756665.026955:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659570688:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756665.026960:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026963:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756665.026965:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756665.026967:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756665.026972:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756665.026976:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756665.026978:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756665.026982:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756665.026984:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756665.026988:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756665.026990:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.026994:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756665.027006:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756665.027010:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756665.027015:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756665.027017:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756665.793472:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756665.793478:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756665.793484:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756666.817433:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756666.817439:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756666.817444:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756667.073706:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756667.073726:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756667.073734:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756667.073742:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756667.073749:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756667.073762:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756667.073772:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756667.073784:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756667.073800:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659570816:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756667.073815:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756667.073824:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756667.073860:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756667.073870:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756667.073883:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.073914:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756667.073924:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133072896 : -123165576478720 : ffff8ffb4790c000) 00000100:00000001:2.0:1787756667.073938:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756667.074003:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756667.074019:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756667.074025:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756667.074028:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.074037:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756667.074051:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.074062:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304394945 00000100:00000001:2.0:1787756667.074108:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756667.074117:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304394945 00000100:00000001:2.0:1787756667.074130:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756667.074168:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756667.074181:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.074239:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756667.074253:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756667/real 1787756667] req@ffff8ffb4790c000 x1874588659570816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756667.074284:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756667.074292:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756667.074303:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756667.074312:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756667.074321:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074329:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756667.074338:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756667.074359:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c000 x1874588659570816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756667.074393:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074405:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756667.074416:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074423:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756667.074431:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074439:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756667.074446:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756667.074458:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756667.074474:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756667.074484:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756667.074500:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659570816:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756667.074517:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074526:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756667.074534:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756667.074541:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756667.074550:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756667.074559:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756667.074566:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756667.074574:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756667.074580:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756667.074587:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756667.074592:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074597:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074613:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756667.074620:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.074629:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756667.074638:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756667.074644:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756667.074649:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756667.074652:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756667.074658:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756667.074661:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756667.074666:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756667.074673:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659570944:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756667.074680:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756667.074684:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756667.074688:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756667.074692:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756667.074698:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.074711:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756667.074714:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133080064 : -123165576471552 : ffff8ffb4790dc00) 00000100:00000001:2.0:1787756667.074721:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756667.074746:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756667.074755:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756667.074759:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756667.074762:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.074765:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756667.074770:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.074777:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304394254 00000100:00000001:2.0:1787756667.074782:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304394945 00000100:00000001:2.0:1787756667.074786:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304394254 00000100:00000001:2.0:1787756667.074790:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304394945 00000100:00000001:2.0:1787756667.074794:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756667.074797:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.074800:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756667.074808:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756667/real 1787756667] req@ffff8ffb4790dc00 x1874588659570944/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756667.074824:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756667.074827:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756667.074832:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756667.074848:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756667.074851:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074855:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756667.074857:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756667.074866:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790dc00 x1874588659570944/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756667.074881:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074885:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756667.074889:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074892:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756667.074895:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.074898:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756667.074918:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756667.074923:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756667.074929:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756667.074934:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756667.074939:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659570944:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756667.074946:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756667.074949:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756667.074952:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756667.074957:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756667.074961:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756667.074967:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756667.074973:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756667.074979:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756667.074982:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756667.074984:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756667.074987:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756667.074992:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075000:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756667.075002:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:2.0:1787756667.075008:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756667.075028:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756667.075035:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756667.075039:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756667.075041:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075043:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756667.075047:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075052:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304394251 00000100:00000001:2.0:1787756667.075056:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304394945 00000100:00000001:2.0:1787756667.075060:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304394251 00000100:00000001:2.0:1787756667.075064:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304394945 00000100:00000001:2.0:1787756667.075095:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756667.075098:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075102:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756667.075109:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756667/real 1787756667] req@ffff8ffb4790f480 x1874588659571072/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756667.075124:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756667.075126:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756667.075131:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756667.075135:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756667.075138:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075141:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756667.075143:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756667.075150:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659571072/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756667.075172:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075176:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756667.075180:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075216:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756667.075218:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075221:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756667.075240:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756667.075245:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756667.075252:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756667.075256:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756667.075263:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756667.075270:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756667.075273:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756667.075275:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756667.075279:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756667.075282:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756667.075285:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756667.075290:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756667.075297:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756667.075300:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756667.075303:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756667.075307:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756667.075312:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075321:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756667.075324:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:2.0:1787756667.075330:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756667.075367:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756667.075374:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756667.075377:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756667.075378:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075381:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756667.075385:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075389:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304394252 00000100:00000001:2.0:1787756667.075392:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304394945 00000100:00000001:2.0:1787756667.075395:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304394252 00000100:00000001:2.0:1787756667.075398:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304394945 00000100:00000001:2.0:1787756667.075400:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756667.075403:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075405:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756667.075410:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756667/real 1787756667] req@ffff8ffb4790ca80 x1874588659571200/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756667.075421:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756667.075423:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756667.075426:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756667.075429:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756667.075430:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075433:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756667.075434:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756667.075439:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659571200/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756667.075447:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075450:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756667.075453:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075455:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756667.075457:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075459:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756667.075473:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756667.075477:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756667.075482:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756667.075484:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756667.075489:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756667.075494:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075497:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756667.075498:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756667.075501:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756667.075504:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756667.075506:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756667.075507:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756667.075511:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756667.075513:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756667.075516:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756667.075518:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075520:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756667.075521:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756667.075523:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756667.075525:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756667.075526:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756667.075528:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756667.075530:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756667.075532:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756667.075534:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756667.075535:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075537:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756667.075539:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756667.075540:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756667.075542:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756667.075543:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756667.075545:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756667.075547:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756667.075548:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756667.075551:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756667.075552:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075555:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756667.075560:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756667.075562:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756667.075565:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756667.075567:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756669.889491:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756669.889501:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756669.889509:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756670.913520:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756670.913526:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756670.913530:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756672.193642:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756672.193654:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756672.193664:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756672.193677:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756672.193687:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756672.193699:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756672.193707:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756672.193718:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756672.193732:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659571328:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756672.193748:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756672.193761:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756672.193772:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756672.193786:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756672.193802:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.193840:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756672.193849:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:2.0:1787756672.193863:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756672.193927:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756672.193948:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756672.193958:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756672.193966:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194003:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756672.194021:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194035:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304400065 00000100:00000001:2.0:1787756672.194060:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756672.194081:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304400065 00000100:00000001:2.0:1787756672.194094:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756672.194120:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756672.194126:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194131:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756672.194141:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756672/real 1787756672] req@ffff8ffb4790ed80 x1874588659571328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756672.194157:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756672.194161:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756672.194172:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756672.194179:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756672.194229:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194233:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756672.194236:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756672.194243:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659571328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756672.194253:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194257:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756672.194261:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194263:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756672.194265:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194268:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756672.194271:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756672.194274:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756672.194279:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756672.194282:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756672.194288:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659571328:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756672.194294:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194297:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756672.194299:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756672.194302:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756672.194307:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756672.194309:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756672.194312:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756672.194317:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756672.194319:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756672.194323:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756672.194325:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194328:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194336:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756672.194339:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194342:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756672.194344:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756672.194346:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756672.194348:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756672.194350:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756672.194353:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756672.194357:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756672.194360:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756672.194364:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571456:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756672.194369:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756672.194371:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756672.194373:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756672.194375:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756672.194380:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194388:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756672.194391:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:2.0:1787756672.194396:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756672.194416:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756672.194423:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756672.194426:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756672.194428:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194430:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756672.194434:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194439:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304399374 00000100:00000001:2.0:1787756672.194443:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304400065 00000100:00000001:2.0:1787756672.194446:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304399374 00000100:00000001:2.0:1787756672.194449:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304400065 00000100:00000001:2.0:1787756672.194452:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756672.194454:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194456:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756672.194462:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756672/real 1787756672] req@ffff8ffb4790fb80 x1874588659571456/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756672.194472:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756672.194473:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756672.194476:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756672.194478:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756672.194480:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194482:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756672.194483:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756672.194489:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659571456/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756672.194498:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194500:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756672.194503:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194505:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756672.194507:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194509:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756672.194524:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756672.194527:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756672.194531:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756672.194534:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756672.194540:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571456:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756672.194547:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756672.194550:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756672.194552:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756672.194557:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756672.194560:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756672.194564:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756672.194570:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756672.194575:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756672.194579:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756672.194582:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756672.194585:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756672.194590:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194601:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756672.194605:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:2.0:1787756672.194612:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756672.194634:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756672.194643:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756672.194647:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756672.194651:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194654:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756672.194660:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194665:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304399371 00000100:00000001:2.0:1787756672.194668:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304400065 00000100:00000001:2.0:1787756672.194671:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304399371 00000100:00000001:2.0:1787756672.194674:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304400065 00000100:00000001:2.0:1787756672.194676:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756672.194679:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194681:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756672.194687:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756672/real 1787756672] req@ffff8ffb4790d180 x1874588659571584/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756672.194697:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756672.194699:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756672.194702:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756672.194704:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756672.194706:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194708:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756672.194710:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756672.194715:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659571584/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756672.194723:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194726:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756672.194729:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194731:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756672.194733:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194735:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756672.194749:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756672.194754:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756672.194761:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756672.194764:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756672.194771:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756672.194779:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756672.194781:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756672.194784:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756672.194788:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756672.194792:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756672.194797:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756672.194803:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756672.194809:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756672.194812:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756672.194816:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756672.194820:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756672.194826:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194837:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756672.194840:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:2.0:1787756672.194848:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756672.194867:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756672.194873:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756672.194876:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756672.194878:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194897:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756672.194901:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194906:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304399371 00000100:00000001:2.0:1787756672.194909:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304400065 00000100:00000001:2.0:1787756672.194917:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304399371 00000100:00000001:2.0:1787756672.194921:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304400065 00000100:00000001:2.0:1787756672.194924:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756672.194927:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.194929:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756672.194935:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756672/real 1787756672] req@ffff8ffb4790f100 x1874588659571712/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756672.194953:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756672.194955:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756672.194958:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756672.194960:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756672.194962:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.194964:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756672.194966:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756672.194971:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659571712/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756672.195006:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.195009:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756672.195012:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.195014:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756672.195016:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.195020:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756672.195040:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756672.195046:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756672.195053:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756672.195057:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756672.195064:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756672.195072:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.195076:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756672.195079:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756672.195082:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756672.195087:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756672.195091:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756672.195093:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756672.195098:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756672.195101:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756672.195105:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756672.195108:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.195112:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756672.195114:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756672.195117:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756672.195120:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756672.195122:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756672.195124:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756672.195127:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756672.195129:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756672.195131:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756672.195133:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.195135:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756672.195137:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756672.195138:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756672.195141:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756672.195142:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756672.195144:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756672.195146:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756672.195147:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756672.195149:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756672.195150:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.195152:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756672.195157:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756672.195159:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756672.195162:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756672.195164:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756673.985492:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756673.985501:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756673.985508:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756675.009478:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756675.009487:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756675.009493:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756677.313756:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756677.313765:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756677.313773:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756677.313779:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756677.313784:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756677.313792:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756677.313800:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756677.313810:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756677.313825:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659571840:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756677.313838:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756677.313847:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756677.313853:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756677.313858:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756677.313865:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.313914:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756677.313922:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787756677.313930:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756677.313986:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756677.314005:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756677.314013:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756677.314020:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314027:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756677.314035:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314045:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304405185 00000100:00000001:2.0:1787756677.314063:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756677.314076:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304405185 00000100:00000001:2.0:1787756677.314088:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756677.314113:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756677.314120:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314130:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756677.314146:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756677/real 1787756677] req@ffff8ffb4a534a80 x1874588659571840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756677.314245:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756677.314250:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756677.314258:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756677.314264:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756677.314268:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314277:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756677.314282:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756677.314299:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659571840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756677.314331:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314344:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756677.314355:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314361:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756677.314367:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314374:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756677.314382:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756677.314389:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.314401:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756677.314409:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756677.314423:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659571840:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756677.314436:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314442:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756677.314447:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756677.314452:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756677.314462:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756677.314469:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756677.314474:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756677.314481:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756677.314486:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756677.314493:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756677.314499:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314509:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314525:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756677.314533:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314544:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756677.314553:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756677.314558:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756677.314566:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756677.314568:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756677.314573:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756677.314580:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756677.314589:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756677.314599:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571968:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756677.314618:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756677.314623:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756677.314628:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756677.314635:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756677.314641:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314653:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756677.314655:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787756677.314661:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756677.314682:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756677.314689:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756677.314692:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756677.314694:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314697:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756677.314700:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314706:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304404494 00000100:00000001:2.0:1787756677.314710:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304405185 00000100:00000001:2.0:1787756677.314713:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304404494 00000100:00000001:2.0:1787756677.314716:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304405185 00000100:00000001:2.0:1787756677.314719:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756677.314721:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314723:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756677.314729:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756677/real 1787756677] req@ffff8ffb4a534700 x1874588659571968/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756677.314739:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756677.314740:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756677.314743:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756677.314745:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756677.314747:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314750:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756677.314751:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756677.314756:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659571968/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756677.314765:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314768:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756677.314770:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314789:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756677.314791:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314793:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756677.314807:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756677.314811:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.314815:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756677.314818:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756677.314823:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659571968:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756677.314828:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756677.314830:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756677.314832:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756677.314835:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756677.314843:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756677.314846:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756677.314850:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756677.314854:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756677.314856:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756677.314858:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756677.314860:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756677.314864:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314870:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756677.314872:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787756677.314877:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756677.314910:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756677.314915:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756677.314919:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756677.314920:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314923:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756677.314926:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314930:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304404491 00000100:00000001:2.0:1787756677.314934:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304405185 00000100:00000001:2.0:1787756677.314937:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304404491 00000100:00000001:2.0:1787756677.314940:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304405185 00000100:00000001:2.0:1787756677.314942:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756677.314945:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.314947:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756677.314953:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756677/real 1787756677] req@ffff8ffb4a535c00 x1874588659572096/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756677.314963:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756677.314965:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756677.314968:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756677.314970:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756677.314972:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314974:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756677.314976:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756677.314981:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659572096/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756677.314989:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314991:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756677.314994:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314996:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756677.314997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.314999:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756677.315011:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756677.315014:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.315017:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756677.315019:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756677.315023:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756677.315027:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315029:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756677.315031:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756677.315034:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756677.315036:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756677.315038:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756677.315040:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756677.315043:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756677.315045:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756677.315047:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756677.315049:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315052:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756677.315053:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756677.315055:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756677.315057:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756677.315059:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756677.315060:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756677.315062:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756677.315063:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756677.315065:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756677.315067:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315069:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315073:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756677.315075:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.315077:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756677.315079:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756677.315080:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756677.315082:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756677.315083:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756677.315086:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756677.315088:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756677.315090:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756677.315094:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756677.315098:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756677.315100:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756677.315102:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756677.315104:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756677.315107:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.315112:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756677.315114:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787756677.315119:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756677.315131:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756677.315136:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756677.315139:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756677.315141:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.315143:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756677.315146:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.315150:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304404491 00000100:00000001:2.0:1787756677.315154:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304405185 00000100:00000001:2.0:1787756677.315157:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304404491 00000100:00000001:2.0:1787756677.315160:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304405185 00000100:00000001:2.0:1787756677.315162:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756677.315165:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.315167:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756677.315171:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756677/real 1787756677] req@ffff8ffb4a537800 x1874588659572224/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756677.315180:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756677.315210:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756677.315214:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756677.315216:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756677.315218:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315220:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756677.315222:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756677.315227:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659572224/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756677.315236:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315239:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756677.315242:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315244:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756677.315245:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315248:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756677.315260:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756677.315263:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.315266:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756677.315269:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756677.315272:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756677.315277:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315280:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756677.315282:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756677.315284:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756677.315286:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756677.315288:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756677.315289:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756677.315292:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756677.315293:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756677.315296:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756677.315298:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315300:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756677.315304:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756677.315306:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756677.315308:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756677.315310:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756678.081445:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756678.081453:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756678.081460:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756679.105480:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756679.105498:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756679.105503:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756682.177321:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756682.177326:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756682.177330:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756682.433531:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756682.433543:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756682.433547:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756682.433551:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756682.433558:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756682.433575:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756682.433582:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756682.433587:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756682.433608:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659572352:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756682.433635:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756682.433646:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756682.433650:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756682.433665:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756682.433677:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.433698:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756682.433704:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787756682.433713:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756682.433774:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756682.433789:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756682.433823:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756682.433827:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.433840:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756682.433847:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.433855:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304410305 00000100:00000001:2.0:1787756682.433864:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756682.433872:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304410305 00000100:00000001:2.0:1787756682.433879:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756682.433894:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756682.433902:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.433904:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756682.433917:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756682/real 1787756682] req@ffff8ffb4a536300 x1874588659572352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756682.433964:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756682.433972:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756682.433984:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756682.433993:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756682.434004:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434014:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756682.434018:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756682.434051:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659572352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756682.434131:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434148:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756682.434165:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434171:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756682.434174:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434178:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756682.434182:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756682.434189:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756682.434266:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756682.434273:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756682.434283:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659572352:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756682.434293:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434298:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756682.434302:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756682.434306:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756682.434340:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756682.434343:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756682.434345:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756682.434350:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756682.434352:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756682.434355:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756682.434357:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434359:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434367:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756682.434369:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434371:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756682.434373:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756682.434375:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756682.434377:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756682.434378:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756682.434381:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756682.434383:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756682.434386:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756682.434390:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572480:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756682.434394:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756682.434396:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756682.434398:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756682.434400:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756682.434403:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434411:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756682.434412:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787756682.434417:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756682.434434:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756682.434439:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756682.434442:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756682.434443:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434444:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756682.434447:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434450:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304409614 00000100:00000001:2.0:1787756682.434453:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304410305 00000100:00000001:2.0:1787756682.434454:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304409614 00000100:00000001:2.0:1787756682.434456:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304410305 00000100:00000001:2.0:1787756682.434458:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756682.434459:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434461:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756682.434465:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756682/real 1787756682] req@ffff8ffb4a537b80 x1874588659572480/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756682.434471:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756682.434472:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756682.434474:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756682.434475:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756682.434476:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434477:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756682.434478:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756682.434480:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659572480/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756682.434485:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434486:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756682.434488:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434489:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756682.434489:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434490:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756682.434500:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756682.434502:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756682.434504:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756682.434506:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756682.434508:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572480:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756682.434511:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756682.434512:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756682.434513:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756682.434515:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756682.434516:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756682.434518:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756682.434520:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756682.434522:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756682.434524:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756682.434525:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756682.434526:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756682.434528:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434531:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756682.434532:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787756682.434535:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756682.434543:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756682.434546:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756682.434547:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756682.434548:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434549:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756682.434551:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434553:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304409611 00000100:00000001:2.0:1787756682.434555:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304410305 00000100:00000001:2.0:1787756682.434557:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304409611 00000100:00000001:2.0:1787756682.434558:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304410305 00000100:00000001:2.0:1787756682.434559:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756682.434561:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434562:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756682.434564:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756682/real 1787756682] req@ffff8ffb4a537800 x1874588659572608/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756682.434568:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756682.434569:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756682.434571:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756682.434572:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756682.434572:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434574:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756682.434574:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756682.434577:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659572608/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756682.434582:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434583:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756682.434585:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434586:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756682.434586:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434587:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756682.434595:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756682.434597:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756682.434598:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756682.434600:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756682.434602:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756682.434604:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756682.434605:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756682.434606:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756682.434607:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756682.434608:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756682.434610:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756682.434612:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572736:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756682.434613:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756682.434615:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756682.434616:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756682.434617:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756682.434619:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434621:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756682.434622:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787756682.434625:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756682.434631:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756682.434633:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756682.434635:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756682.434636:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434637:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756682.434639:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434641:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304409611 00000100:00000001:2.0:1787756682.434642:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304410305 00000100:00000001:2.0:1787756682.434644:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304409611 00000100:00000001:2.0:1787756682.434646:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304410305 00000100:00000001:2.0:1787756682.434647:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756682.434649:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434650:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756682.434652:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756682/real 1787756682] req@ffff8ffb4a535c00 x1874588659572736/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756682.434656:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756682.434657:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756682.434658:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756682.434659:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756682.434660:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434661:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756682.434661:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756682.434664:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659572736/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756682.434668:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434669:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756682.434671:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434672:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756682.434673:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434674:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756682.434683:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756682.434684:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756682.434686:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756682.434687:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756682.434689:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659572736:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756682.434692:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434693:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756682.434694:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756682.434695:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756682.434696:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756682.434697:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756682.434698:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756682.434700:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756682.434700:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756682.434702:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756682.434703:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434704:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756682.434705:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756682.434706:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756682.434707:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756682.434707:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756682.434708:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756682.434710:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756682.434710:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756682.434711:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756682.434712:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434713:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756682.434714:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756682.434714:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756682.434715:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756682.434716:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756682.434716:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756682.434718:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756682.434718:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756682.434719:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756682.434720:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434721:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756682.434723:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756682.434724:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756682.434725:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756682.434726:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756683.201479:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756683.201489:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756683.201496:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756686.273489:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756686.273495:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756686.273500:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756687.297316:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756687.297320:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756687.297322:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756687.553365:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756687.553383:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756687.553387:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756687.553393:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756687.553400:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756687.553412:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9742 00000100:00080000:2.0:1787756687.553441:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756687.553446:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756687.553449:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756687.553455:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756687.553460:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756687.553465:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756687.553468:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756687.553471:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756687.553473:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756687.553492:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756687.553495:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756687.553499:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756687.553505:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756687.553510:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.553512:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756687.553516:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756687.553534:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756687.553536:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.553541:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756687.553545:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756687.553552:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.553571:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756687.553586:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659572864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756687.553616:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756687.553625:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.553666:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756687.553674:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756687.553686:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756687.553692:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756687.553695:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756687.553703:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756687.553735:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9742 00000100:00080000:2.0:1787756687.553742:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756687.553756:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756687.553758:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756687.553766:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756687.553782:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756687.553787:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756687.553795:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756687.553812:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756687.553833:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756687.553855:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756687.553873:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756687.553886:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756687.553906:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756687.553928:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.553933:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756687.553941:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756687.553946:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.553950:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756687.553964:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756687.553969:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.553972:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756687.553975:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756687.553990:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554012:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756687.554026:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659572992/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756687.554063:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756687.554068:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554080:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756687.554086:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756687.554097:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756687.554100:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756687.554103:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756687.554117:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756687.554128:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9742 00000100:00080000:2.0:1787756687.554146:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756687.554157:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756687.554166:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756687.554180:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756687.554261:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756687.554267:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756687.554279:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756687.554295:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756687.554299:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756687.554312:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756687.554319:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756687.554334:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756687.554343:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756687.554353:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554360:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756687.554368:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756687.554384:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554396:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756687.554411:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756687.554428:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554443:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756687.554449:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756687.554455:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554472:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756687.554506:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659573120/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756687.554540:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756687.554548:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554557:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756687.554571:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756687.554578:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756687.554583:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756687.554593:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756687.554600:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756687.554623:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9742 00000100:00080000:2.0:1787756687.554644:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756687.554655:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756687.554670:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756687.554682:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756687.554690:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756687.554703:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756687.554738:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756687.554755:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756687.554762:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756687.554783:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756687.554801:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756687.554810:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756687.554823:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756687.554844:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554859:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756687.554870:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756687.554883:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554899:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756687.554916:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756687.554918:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554923:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756687.554926:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756687.554929:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.554951:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756687.554978:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659573248/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756687.555011:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756687.555020:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756687.555031:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756690.369431:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756690.369436:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756690.369440:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756691.393529:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756691.393535:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756691.393541:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756692.673340:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756692.673369:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756692.673372:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756692.673375:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756692.673378:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756692.673385:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9747 00000100:00080000:3.0:1787756692.673395:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756692.673398:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756692.673400:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756692.673404:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756692.673407:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756692.673409:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756692.673411:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756692.673413:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756692.673414:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756692.673417:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756692.673419:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756692.673421:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756692.673423:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756692.673426:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673428:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756692.673431:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756692.673440:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756692.673441:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673444:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756692.673446:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756692.673449:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673459:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756692.673467:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659573376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756692.673475:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756692.673489:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673502:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756692.673505:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756692.673507:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756692.673509:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756692.673510:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756692.673512:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756692.673514:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9747 00000100:00080000:3.0:1787756692.673516:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756692.673517:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756692.673519:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756692.673521:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756692.673522:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756692.673523:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756692.673525:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756692.673527:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756692.673528:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756692.673530:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756692.673531:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756692.673532:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756692.673536:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756692.673538:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673539:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756692.673543:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756692.673545:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673546:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756692.673548:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756692.673549:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673551:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756692.673553:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756692.673555:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673561:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756692.673567:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659573504/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756692.673579:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756692.673586:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673596:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756692.673605:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756692.673625:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756692.673640:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756692.673645:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756692.673652:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756692.673658:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9747 00000100:00080000:3.0:1787756692.673664:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756692.673667:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756692.673668:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756692.673670:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756692.673671:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756692.673673:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756692.673676:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756692.673679:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756692.673684:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756692.673689:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756692.673692:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756692.673693:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756692.673696:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756692.673698:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673700:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756692.673702:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756692.673704:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673705:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756692.673709:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756692.673710:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673712:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756692.673713:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756692.673714:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673720:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756692.673724:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659573632/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756692.673730:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756692.673734:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673740:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756692.673743:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756692.673746:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756692.673749:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756692.673752:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756692.673755:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756692.673760:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9747 00000100:00080000:3.0:1787756692.673767:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756692.673772:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756692.673774:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756692.673778:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756692.673781:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756692.673784:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756692.673786:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756692.673788:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756692.673789:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756692.673792:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756692.673796:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756692.673798:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756692.673803:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756692.673806:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673810:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756692.673813:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756692.673816:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673820:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756692.673826:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756692.673828:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673833:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756692.673835:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756692.673837:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673843:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756692.673852:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659573760/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756692.673889:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756692.673897:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756692.673906:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756694.465480:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756694.465486:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756694.465491:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756695.489419:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756695.489424:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756695.489428:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756697.793658:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756697.793668:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756697.793675:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756697.793682:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756697.793691:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756697.793701:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756697.793710:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756697.793719:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756697.793731:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659573888:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756697.793746:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756697.793756:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756697.793762:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756697.793768:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756697.793777:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.793805:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756697.793812:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:2.0:1787756697.793822:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756697.793875:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756697.793896:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756697.793922:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756697.793932:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.793939:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756697.793952:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.793963:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304425665 00000100:00000001:2.0:1787756697.793984:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756697.793997:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304425665 00000100:00000001:2.0:1787756697.794011:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756697.794021:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756697.794030:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794040:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756697.794057:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756697/real 1787756697] req@ffff8ffb4790c380 x1874588659573888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756697.794098:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756697.794104:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756697.794115:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756697.794122:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756697.794130:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794141:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756697.794147:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756697.794167:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659573888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756697.794242:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794253:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756697.794260:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794264:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756697.794271:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794281:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756697.794287:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756697.794299:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756697.794316:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756697.794328:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756697.794340:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659573888:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756697.794349:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794353:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756697.794355:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756697.794358:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756697.794364:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756697.794367:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756697.794369:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756697.794375:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756697.794378:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756697.794381:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756697.794384:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794387:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794396:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756697.794398:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794401:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756697.794403:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756697.794406:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756697.794410:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756697.794412:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756697.794416:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756697.794419:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756697.794422:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756697.794428:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574016:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756697.794433:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756697.794436:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756697.794439:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756697.794443:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756697.794449:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794460:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756697.794463:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:2.0:1787756697.794470:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756697.794493:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756697.794499:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756697.794503:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756697.794504:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794506:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756697.794510:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794515:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304424974 00000100:00000001:2.0:1787756697.794519:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304425665 00000100:00000001:2.0:1787756697.794552:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304424974 00000100:00000001:2.0:1787756697.794556:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304425665 00000100:00000001:2.0:1787756697.794559:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756697.794562:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794565:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756697.794572:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756697/real 1787756697] req@ffff8ffb4790d880 x1874588659574016/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756697.794587:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756697.794589:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756697.794596:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756697.794600:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756697.794604:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794607:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756697.794608:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756697.794614:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659574016/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756697.794626:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794630:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756697.794636:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794642:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756697.794645:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794648:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756697.794673:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756697.794678:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756697.794683:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756697.794688:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756697.794694:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574016:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756697.794702:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756697.794706:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756697.794710:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756697.794713:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756697.794717:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756697.794720:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756697.794726:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756697.794730:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756697.794734:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756697.794736:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756697.794738:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756697.794743:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794751:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756697.794753:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:2.0:1787756697.794758:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756697.794775:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756697.794783:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756697.794786:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756697.794789:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794791:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756697.794796:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794801:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304424971 00000100:00000001:2.0:1787756697.794805:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304425665 00000100:00000001:2.0:1787756697.794809:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304424971 00000100:00000001:2.0:1787756697.794813:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304425665 00000100:00000001:2.0:1787756697.794816:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756697.794821:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794823:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756697.794828:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756697/real 1787756697] req@ffff8ffb4790e680 x1874588659574144/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756697.794838:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756697.794841:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756697.794848:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756697.794852:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756697.794854:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794857:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756697.794860:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756697.794866:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659574144/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756697.794876:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794879:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756697.794882:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794885:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756697.794886:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.794888:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756697.794910:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756697.794913:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756697.794919:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756697.794923:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756697.794930:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756697.794941:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756697.794945:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756697.794948:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756697.794952:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756697.794956:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756697.794960:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756697.794967:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574272:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756697.794971:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756697.794977:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756697.794980:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756697.794984:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756697.794987:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.794998:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756697.795000:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:2.0:1787756697.795005:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756697.795018:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756697.795024:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756697.795027:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756697.795032:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.795035:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756697.795043:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.795047:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304424971 00000100:00000001:2.0:1787756697.795051:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304425665 00000100:00000001:2.0:1787756697.795054:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304424971 00000100:00000001:2.0:1787756697.795057:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304425665 00000100:00000001:2.0:1787756697.795060:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756697.795062:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.795065:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756697.795071:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756697/real 1787756697] req@ffff8ffb4790ce00 x1874588659574272/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756697.795083:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756697.795085:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756697.795090:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756697.795095:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756697.795103:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795106:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756697.795110:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756697.795118:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659574272/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756697.795129:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795132:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756697.795137:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795141:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756697.795145:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795148:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756697.795166:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756697.795170:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756697.795174:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756697.795177:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756697.795212:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574272:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756697.795220:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795224:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756697.795226:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756697.795228:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756697.795232:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756697.795234:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756697.795236:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756697.795240:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756697.795241:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756697.795245:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756697.795248:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795250:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756697.795253:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756697.795256:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756697.795260:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756697.795262:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756697.795264:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756697.795267:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756697.795269:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756697.795275:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756697.795279:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795284:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756697.795287:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756697.795293:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756697.795296:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756697.795297:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756697.795302:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756697.795308:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756697.795311:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756697.795315:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756697.795317:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795320:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756697.795326:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756697.795328:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756697.795330:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756697.795333:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756698.561413:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756698.561422:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756698.561428:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756699.585558:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756699.585564:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756699.585567:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756702.657520:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756702.657526:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756702.657531:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756702.913362:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756702.913378:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756702.913382:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756702.913388:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756702.913395:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756702.913416:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9757 00000100:00080000:3.0:1787756702.913454:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756702.913461:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756702.913465:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756702.913472:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756702.913476:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756702.913481:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756702.913484:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756702.913487:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756702.913489:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756702.913497:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756702.913501:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756702.913506:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756702.913511:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756702.913516:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913519:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756702.913524:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756702.913541:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756702.913544:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913549:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756702.913553:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756702.913560:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913582:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756702.913603:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659574400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756702.913625:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756702.913635:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913664:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756702.913671:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756702.913678:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756702.913681:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756702.913685:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756702.913692:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756702.913698:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9757 00000100:00080000:3.0:1787756702.913708:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756702.913715:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756702.913720:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756702.913725:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787756702.913729:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:3.0:1787756702.913730:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787756702.913740:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:3.0:1787756702.913741:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756702.913747:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:3.0:1787756702.913752:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756702.913753:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756702.913759:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913764:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787756702.913768:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787756702.913772:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787756702.913775:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756702.913783:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756702.913789:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1787756702.913797:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659574400:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756702.913803:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.913807:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756702.913815:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756702.913817:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756702.913824:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.913827:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756702.913831:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756702.913837:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756702.913841:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913849:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756702.913859:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756702.913866:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00000001:3.0:1787756702.913867:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.913872:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:3.0:1787756702.913878:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.913881:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787756702.913896:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756702.913901:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913906:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756702.913909:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756702.913914:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913926:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756702.913942:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659574528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756702.913953:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756702.913972:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756702.913976:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756702.913979:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.913984:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756702.913990:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.913994:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.913995:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787756702.914006:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787756702.914007:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914018:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304430785 00000100:00080000:3.0:1787756702.914019:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756702.914026:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756702.914030:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756702.914035:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787756702.914039:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787756702.914043:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9757 00000100:00000001:2.0:1787756702.914052:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304430785 00000100:00080000:3.0:1787756702.914055:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756702.914062:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756702.914063:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756702.914065:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756702.914074:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787756702.914074:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756702.914081:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787756702.914083:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756702.914087:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756702.914090:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:3.0:1787756702.914094:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756702.914103:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756702.914106:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787756702.914107:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756702/real 1787756702] req@ffff8ffb45c30e00 x1874588659574400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756702.914115:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756702.914125:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756702.914133:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756702.914141:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756702.914147:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.914150:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000400:00000001:2.0:1787756702.914151:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000008:00100000:3.0:1787756702.914155:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:2.0:1787756702.914159:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000008:00000001:3.0:1787756702.914162:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.914168:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914173:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787756702.914176:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756702.914181:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914227:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756702.914230:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787756702.914234:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787756702.914236:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756702.914243:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787756702.914244:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914252:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756702.914257:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756702.914271:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659574400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756702.914272:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659574656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756702.914331:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756702.914333:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756702.914336:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.914347:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914348:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756702.914351:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756702.914361:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756702.914362:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756702.914364:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756702.914367:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756702.914370:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9757 00000100:00000001:2.0:1787756702.914373:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787756702.914376:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756702.914381:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756702.914384:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756702.914386:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756702.914389:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756702.914392:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000020:00000001:3.0:1787756702.914398:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787756702.914400:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787756702.914406:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00080000:2.0:1787756702.914408:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000020:00000001:3.0:1787756702.914417:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:2.0:1787756702.914420:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756702.914431:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:3.0:1787756702.914431:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756702.914469:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:2.0:1787756702.914471:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659574400:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756702.914478:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756702.914484:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.914491:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756702.914497:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00080000:3.0:1787756702.914497:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787756702.914504:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787756702.914512:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.914515:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:3.0:1787756702.914522:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000020:00000001:2.0:1787756702.914523:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756702.914530:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756702.914535:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756702.914538:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756702.914543:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 02000000:00000001:3.0:1787756702.914549:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914550:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756702.914555:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756702.914560:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756702.914561:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:3.0:1787756702.914570:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756702.914577:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000008:00000001:3.0:1787756702.914580:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914583:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.914589:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914590:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756702.914596:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756702.914600:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756702.914606:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787756702.914606:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787756702.914615:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756702.914616:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.914619:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756702.914626:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756702.914627:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756702.914632:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756702.914635:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:2.0:1787756702.914643:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574528:192.168.201.118@tcp:38:kworker.0 02000000:00000001:3.0:1787756702.914646:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914650:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756702.914657:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756702.914659:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.914662:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787756702.914664:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.914668:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914683:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:3.0:1787756702.914683:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659574784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756702.914688:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787756702.914700:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787756702.914711:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756702.914717:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756702.914725:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787756702.914727:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756702.914740:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756702.914745:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756702.914748:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914750:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756702.914754:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914760:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304430094 00000100:00000001:2.0:1787756702.914765:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304430785 00000100:00000001:2.0:1787756702.914768:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304430094 00000100:00000001:2.0:1787756702.914773:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304430785 00000100:00000001:2.0:1787756702.914776:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756702.914780:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914782:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756702.914789:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756702/real 1787756702] req@ffff8ffb45c33800 x1874588659574528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756702.914802:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756702.914804:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756702.914807:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756702.914810:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756702.914813:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.914816:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756702.914818:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756702.914822:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659574528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756702.914831:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.914834:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756702.914837:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.914839:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756702.914840:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.914842:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756702.914864:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756702.914867:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756702.914871:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756702.914873:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756702.914877:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574528:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756702.914882:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756702.914884:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756702.914886:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.914889:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756702.914891:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756702.914894:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756702.914898:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756702.914902:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756702.914904:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756702.914906:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.914908:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756702.914912:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914918:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756702.914920:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787756702.914925:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756702.914939:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756702.914944:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756702.914946:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756702.914949:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914951:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756702.914954:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914958:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304430091 00000100:00000001:2.0:1787756702.914962:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304430785 00000100:00000001:2.0:1787756702.914965:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304430091 00000100:00000001:2.0:1787756702.914968:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304430785 00000100:00000001:2.0:1787756702.914971:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756702.914973:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.914975:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756702.914980:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756702/real 1787756702] req@ffff8ffb45c30700 x1874588659574656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756702.914989:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756702.914990:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756702.914992:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756702.914995:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756702.914997:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.914999:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756702.915001:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756702.915005:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659574656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756702.915014:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915016:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756702.915019:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915021:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756702.915022:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915024:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756702.915043:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756702.915046:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756702.915050:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756702.915053:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756702.915057:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756702.915062:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915064:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756702.915065:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756702.915067:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756702.915070:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756702.915072:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756702.915073:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756702.915076:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756702.915077:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756702.915080:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756702.915082:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915084:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756702.915085:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756702.915087:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756702.915089:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756702.915090:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756702.915092:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756702.915094:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756702.915095:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756702.915097:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756702.915098:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915101:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915107:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756702.915109:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.915113:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756702.915115:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756702.915117:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756702.915120:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756702.915123:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.915128:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756702.915131:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756702.915134:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756702.915141:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574784:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756702.915148:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756702.915152:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756702.915155:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756702.915159:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756702.915164:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.915174:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756702.915177:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787756702.915222:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756702.915245:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756702.915253:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756702.915258:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756702.915262:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.915265:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756702.915271:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.915277:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304430092 00000100:00000001:2.0:1787756702.915282:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304430785 00000100:00000001:2.0:1787756702.915286:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304430092 00000100:00000001:2.0:1787756702.915291:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304430785 00000100:00000001:2.0:1787756702.915295:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756702.915299:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.915303:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756702.915312:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756702/real 1787756702] req@ffff8ffb45c32a00 x1874588659574784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756702.915330:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756702.915333:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756702.915338:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756702.915341:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756702.915344:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915348:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756702.915350:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756702.915357:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659574784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756702.915366:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915369:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756702.915373:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915375:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756702.915377:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915379:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756702.915399:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756702.915403:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756702.915407:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756702.915410:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756702.915414:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659574784:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756702.915419:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915421:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756702.915423:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756702.915425:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756702.915428:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756702.915430:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756702.915432:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756702.915462:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756702.915464:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756702.915467:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756702.915469:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915472:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756702.915477:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756702.915478:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756702.915481:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756702.915484:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756703.681551:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756703.681560:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756703.681565:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756706.753593:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756706.753601:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756706.753608:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756707.777518:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756707.777527:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756707.777534:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756708.033715:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756708.033726:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756708.033734:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756708.033741:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756708.033749:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756708.033762:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756708.033768:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756708.033774:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756708.033794:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659574912:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756708.033816:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756708.033831:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756708.033840:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756708.033856:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756708.033874:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.033912:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756708.033939:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133078272 : -123165576473344 : ffff8ffb4790d500) 00000100:00000001:2.0:1787756708.033957:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756708.034047:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756708.034081:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756708.034095:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756708.034107:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034132:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756708.034152:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034163:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304435905 00000100:00000001:2.0:1787756708.034182:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756708.034228:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304435906 00000100:00000001:2.0:1787756708.034235:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756708.034240:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756708.034243:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034248:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756708.034258:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756708/real 1787756708] req@ffff8ffb4790d500 x1874588659574912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756708.034274:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756708.034276:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756708.034282:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756708.034286:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756708.034288:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034291:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756708.034294:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756708.034300:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d500 x1874588659574912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756708.034310:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034315:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756708.034320:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034323:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756708.034325:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034328:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756708.034331:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756708.034336:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756708.034341:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756708.034373:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756708.034379:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659574912:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756708.034386:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034390:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756708.034393:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756708.034397:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756708.034404:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756708.034407:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756708.034410:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756708.034418:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756708.034420:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756708.034424:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756708.034426:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034429:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034440:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756708.034443:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034446:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756708.034448:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756708.034450:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756708.034452:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756708.034455:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756708.034458:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756708.034461:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756708.034464:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756708.034468:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659575040:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756708.034473:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756708.034476:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756708.034478:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756708.034481:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756708.034485:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034495:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756708.034497:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133087232 : -123165576464384 : ffff8ffb4790f800) 00000100:00000001:2.0:1787756708.034504:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756708.034523:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756708.034530:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756708.034533:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756708.034535:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034537:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756708.034541:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034547:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304435214 00000100:00000001:2.0:1787756708.034550:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304435905 00000100:00000001:2.0:1787756708.034553:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304435214 00000100:00000001:2.0:1787756708.034557:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304435906 00000100:00000001:2.0:1787756708.034559:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756708.034562:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034564:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756708.034569:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756708/real 1787756708] req@ffff8ffb4790f800 x1874588659575040/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756708.034579:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756708.034581:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756708.034583:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756708.034585:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756708.034587:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034590:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756708.034592:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756708.034596:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f800 x1874588659575040/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756708.034605:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034608:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756708.034611:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034612:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756708.034615:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034617:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756708.034639:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756708.034643:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756708.034647:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756708.034650:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756708.034655:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659575040:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756708.034659:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756708.034661:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756708.034663:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756708.034665:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756708.034667:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756708.034670:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756708.034673:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659575168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756708.034678:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756708.034680:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756708.034682:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756708.034684:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756708.034688:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034694:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756708.034696:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133081856 : -123165576469760 : ffff8ffb4790e300) 00000100:00000001:2.0:1787756708.034701:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756708.034713:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756708.034718:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756708.034720:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756708.034722:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034724:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756708.034727:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034731:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304435211 00000100:00000001:2.0:1787756708.034734:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304435905 00000100:00000001:2.0:1787756708.034737:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304435211 00000100:00000001:2.0:1787756708.034740:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304435906 00000100:00000001:2.0:1787756708.034743:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756708.034746:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034748:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756708.034753:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756708/real 1787756708] req@ffff8ffb4790e300 x1874588659575168/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756708.034763:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756708.034764:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756708.034767:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756708.034769:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756708.034770:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034773:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756708.034775:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756708.034779:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e300 x1874588659575168/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756708.034787:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034790:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756708.034793:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034795:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756708.034797:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034799:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756708.034817:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756708.034820:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756708.034824:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756708.034827:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756708.034831:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659575168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756708.034836:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756708.034837:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756708.034839:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756708.034841:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756708.034843:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756708.034846:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756708.034850:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659575296:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756708.034854:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756708.034856:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756708.034858:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756708.034860:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756708.034864:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034869:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756708.034870:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:2.0:1787756708.034875:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756708.034886:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756708.034890:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756708.034893:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756708.034894:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034896:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756708.034899:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034903:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304435211 00000100:00000001:2.0:1787756708.034906:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304435905 00000100:00000001:2.0:1787756708.034909:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304435211 00000100:00000001:2.0:1787756708.034912:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304435906 00000100:00000001:2.0:1787756708.034915:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756708.034917:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.034919:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756708.034924:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756708/real 1787756708] req@ffff8ffb4790ce00 x1874588659575296/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756708.034932:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756708.034934:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756708.034936:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756708.034938:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756708.034939:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034941:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756708.034943:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756708.034947:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659575296/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756708.034964:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034967:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756708.034970:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034972:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756708.034974:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.034976:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756708.034999:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756708.035005:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756708.035010:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756708.035014:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756708.035020:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659575296:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756708.035027:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.035030:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756708.035033:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756708.035037:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756708.035042:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756708.035045:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756708.035048:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756708.035053:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756708.035055:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756708.035059:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756708.035062:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.035066:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756708.035068:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756708.035071:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756708.035074:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756708.035076:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756708.035079:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756708.035082:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756708.035084:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756708.035088:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756708.035090:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.035095:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756708.035096:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756708.035099:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756708.035101:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756708.035103:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756708.035105:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756708.035107:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756708.035109:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756708.035111:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756708.035113:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.035115:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756708.035120:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756708.035122:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756708.035124:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756708.035127:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756710.849496:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756710.849503:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756710.849508:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756711.873574:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756711.873583:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756711.873589:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756713.153293:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756713.153300:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756713.153302:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756713.153305:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756713.153308:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756713.153314:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9767 00000100:00080000:2.0:1787756713.153322:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756713.153325:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756713.153326:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756713.153329:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756713.153331:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756713.153334:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756713.153335:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756713.153337:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756713.153338:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756713.153341:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756713.153343:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756713.153345:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756713.153348:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756713.153350:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153351:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756713.153353:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756713.153362:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756713.153363:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153365:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756713.153367:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756713.153371:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153380:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756713.153387:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659575424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756713.153395:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756713.153398:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153412:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756713.153415:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756713.153419:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756713.153421:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756713.153422:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756713.153426:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756713.153428:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9767 00000100:00080000:2.0:1787756713.153432:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756713.153434:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756713.153436:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756713.153438:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756713.153440:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756713.153442:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756713.153444:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756713.153446:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756713.153448:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756713.153451:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756713.153453:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756713.153455:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756713.153460:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756713.153462:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153465:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756713.153469:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756713.153472:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153474:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756713.153480:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756713.153481:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153483:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756713.153484:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756713.153486:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153500:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756713.153506:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659575552/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756713.153514:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756713.153516:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153518:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756713.153521:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756713.153524:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756713.153535:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756713.153536:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756713.153539:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756713.153541:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9767 00000100:00080000:2.0:1787756713.153551:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756713.153553:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756713.153555:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756713.153558:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756713.153559:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756713.153561:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756713.153563:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756713.153565:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756713.153566:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756713.153569:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756713.153572:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756713.153573:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756713.153576:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756713.153579:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153581:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756713.153583:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756713.153586:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153587:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756713.153591:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756713.153592:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153594:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756713.153595:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756713.153597:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153601:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756713.153607:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659575680/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756713.153615:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756713.153617:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153619:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756713.153621:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756713.153625:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756713.153626:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756713.153628:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756713.153631:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756713.153633:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9767 00000100:00080000:2.0:1787756713.153637:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756713.153639:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756713.153640:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756713.153642:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756713.153644:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756713.153646:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756713.153648:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756713.153651:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756713.153652:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756713.153655:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756713.153658:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756713.153660:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756713.153663:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756713.153665:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153667:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756713.153669:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756713.153673:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153674:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756713.153678:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756713.153680:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153683:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756713.153684:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756713.153686:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153691:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756713.153695:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659575808/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756713.153710:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756713.153712:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756713.153714:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756714.945399:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756714.945403:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756714.945407:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756715.969557:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756715.969569:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756715.969578:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756718.273309:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756718.273322:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756718.273326:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756718.273330:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756718.273336:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756718.273345:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9773 00000100:00080000:2.0:1787756718.273358:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756718.273362:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756718.273365:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756718.273370:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756718.273374:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756718.273378:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756718.273381:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756718.273384:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756718.273385:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756718.273391:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756718.273394:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756718.273398:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756718.273402:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756718.273406:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273408:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756718.273411:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756718.273424:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756718.273426:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273430:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756718.273434:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756718.273439:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273454:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756718.273466:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659575936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756718.273480:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756718.273485:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273507:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756718.273512:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756718.273516:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756718.273518:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756718.273521:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756718.273525:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756718.273528:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9773 00000100:00080000:2.0:1787756718.273532:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756718.273534:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756718.273536:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756718.273540:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756718.273542:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756718.273544:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756718.273547:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756718.273550:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756718.273551:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756718.273555:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756718.273558:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756718.273561:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756718.273568:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756718.273572:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273580:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756718.273587:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756718.273594:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273599:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756718.273607:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756718.273610:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273615:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756718.273625:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756718.273636:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273666:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756718.273688:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659576064/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756718.273709:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756718.273718:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273730:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756718.273741:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756718.273749:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756718.273768:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756718.273776:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756718.273786:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756718.273793:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9773 00000100:00080000:2.0:1787756718.273806:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756718.273812:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756718.273815:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756718.273832:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756718.273840:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756718.273849:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756718.273857:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756718.273867:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756718.273877:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756718.273896:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756718.273910:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756718.273917:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756718.273926:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756718.273935:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273939:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756718.273950:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756718.273963:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.273974:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756718.273988:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756718.273994:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.274005:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756718.274012:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756718.274023:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.274043:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756718.274066:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659576192/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756718.274106:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756718.274114:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.274121:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756718.274132:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756718.274144:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756718.274152:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756718.274159:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756718.274233:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756718.274239:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9773 00000100:00080000:2.0:1787756718.274245:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756718.274253:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756718.274259:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756718.274269:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756718.274275:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756718.274279:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756718.274286:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756718.274293:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756718.274299:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756718.274310:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756718.274315:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756718.274322:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756718.274329:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756718.274337:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.274344:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756718.274351:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756718.274360:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.274369:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756718.274386:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756718.274394:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.274402:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756718.274409:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756718.274418:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.274430:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756718.274449:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659576320/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756718.274470:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756718.274477:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756718.274488:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756719.041338:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756719.041342:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756719.041346:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756720.065471:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756720.065479:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756720.065485:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756723.137334:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756723.137337:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756723.137340:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756723.393454:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756723.395239:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756723.395245:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756723.395250:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756723.395258:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756723.395269:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9778 00000100:00080000:3.0:1787756723.396985:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756723.396993:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756723.396996:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756723.397020:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756723.397024:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756723.397031:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756723.397043:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756723.397046:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756723.397048:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756723.397057:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756723.397060:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756723.397064:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756723.397070:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756723.397099:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397103:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756723.397107:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756723.397128:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756723.397131:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397136:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756723.397140:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756723.397147:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397166:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756723.397181:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659576448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756723.397234:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756723.397242:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397280:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756723.397286:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756723.397291:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756723.397293:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756723.397296:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756723.397321:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756723.397325:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9778 00000100:00080000:3.0:1787756723.397343:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756723.397347:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756723.397349:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756723.397353:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756723.397357:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756723.397361:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756723.397364:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756723.397367:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756723.397369:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756723.397374:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756723.397378:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756723.397382:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756723.397395:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756723.397402:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397408:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756723.397414:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756723.397421:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397429:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756723.397437:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756723.397442:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397445:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756723.397450:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756723.397454:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397462:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756723.397469:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659576576/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756723.397489:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756723.397495:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397505:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756723.397512:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756723.397518:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756723.397520:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756723.397523:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756723.397532:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756723.397542:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9778 00000100:00080000:3.0:1787756723.397548:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756723.397553:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756723.397555:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756723.397558:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756723.397559:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756723.397562:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756723.397565:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756723.397567:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756723.397569:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756723.397573:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756723.397575:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756723.397577:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756723.397581:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756723.397583:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397585:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756723.397588:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756723.397591:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397593:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756723.397599:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756723.397603:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397609:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756723.397610:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756723.397612:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397622:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756723.397628:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659576704/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756723.397642:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756723.397650:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397657:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756723.397662:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756723.397667:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756723.397669:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756723.397671:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756723.397676:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756723.397680:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9778 00000100:00080000:3.0:1787756723.397685:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756723.397687:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756723.397689:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756723.397691:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756723.397692:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756723.397695:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756723.397698:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756723.397702:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756723.397705:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756723.397708:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756723.397712:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756723.397722:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756723.397726:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756723.397728:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397731:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756723.397734:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756723.397740:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397746:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756723.397754:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756723.397756:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397759:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756723.397761:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756723.397764:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397775:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756723.397790:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659576832/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756723.397804:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756723.397810:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756723.397820:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756724.161329:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756724.161334:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756724.161339:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756727.233484:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756727.233525:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756727.233532:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756728.257509:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756728.257516:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756728.257521:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756728.513570:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756728.513578:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756728.513582:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756728.513587:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756728.513590:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756728.513597:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756728.513602:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756728.513607:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756728.513621:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659576960:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756728.513629:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756728.513635:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756728.513639:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756728.513642:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756728.513649:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.513669:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756728.513673:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:2.0:1787756728.513679:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756728.513745:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.513756:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304456385 00000100:00000001:2.0:1787756728.513770:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756728.513782:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304456385 00000100:00000001:2.0:1787756728.513791:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756728.513798:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756728.513814:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.513821:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.513841:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756728.513849:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756728.513853:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787756728.513860:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756728.513863:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756728.513867:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756728.513870:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756728.513873:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756728.513879:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756728.513884:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756728.513888:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756728.513898:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577088:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756728.513907:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756728.513911:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756728.513914:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756728.513920:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756728.513926:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.513938:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756728.513943:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:2.0:1787756728.513952:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756728.513974:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514029:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304455693 00000100:00000001:2.0:1787756728.514038:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304456385 00000100:00000001:2.0:1787756728.514046:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304455693 00000100:00000001:2.0:1787756728.514054:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304456385 00000100:00000001:2.0:1787756728.514064:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756728.514071:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514076:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514080:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756728.514083:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756728.514084:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787756728.514087:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756728.514089:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756728.514091:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756728.514092:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756728.514094:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756728.514097:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756728.514099:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756728.514102:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756728.514108:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756728.514111:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756728.514113:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756728.514115:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756728.514116:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756728.514120:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514132:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756728.514133:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:2.0:1787756728.514137:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756728.514150:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514153:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304455690 00000100:00000001:2.0:1787756728.514156:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304456385 00000100:00000001:2.0:1787756728.514158:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304455690 00000100:00000001:2.0:1787756728.514161:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304456385 00000100:00000001:2.0:1787756728.514163:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756728.514165:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514166:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756728.514167:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756728.514168:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756728.514170:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756728.514171:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756728.514172:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756728.514176:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577344:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756728.514179:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756728.514180:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756728.514181:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756728.514182:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756728.514228:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514233:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756728.514234:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:2.0:1787756728.514248:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756728.514261:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514264:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304455691 00000100:00000001:2.0:1787756728.514267:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304456385 00000100:00000001:2.0:1787756728.514270:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304455691 00000100:00000001:2.0:1787756728.514274:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304456385 00000100:00000001:2.0:1787756728.514278:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756728.514280:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514282:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514284:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756728.514289:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756728.514290:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787756728.514293:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756728.514294:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756728.514296:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756728.514298:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756731.329489:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756731.329497:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756731.329502:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787756731.585613:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787756731.585660:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787756731.585677:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756731.585692:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756731.585711:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756731.585714:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756731.585716:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756731.585725:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756731.585729:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756731.585731:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756731.585732:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756731.585734:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756731.585738:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756731.585742:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756731.585745:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756731.585747:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756731.585749:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756731.585755:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756731.585762:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756731.585765:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756731.585768:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756731.585771:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756731.585773:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756731.585780:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756731.585784:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756731.585798:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756728/real 1787756731] req@ffff8ffb4790c380 x1874588659577216/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756731.585817:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756731.585819:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756731.585827:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756731.585831:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756731.585833:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.585838:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756731.585840:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756731.585848:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659577216/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756731.585860:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.585865:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756731.585870:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.585872:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756731.585875:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.585879:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756731.585897:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756731.585902:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756731.585908:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756731.585913:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756731.585918:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756731.585924:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756731.585950:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756728/real 1787756731] req@ffff8ffb4790f100 x1874588659577344/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756731.585961:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756731.585963:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756731.585969:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756731.585971:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756731.585973:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.585975:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756731.585977:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756731.585989:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659577344/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756731.585999:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586001:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756731.586004:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586006:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756731.586008:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586010:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756731.586027:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756731.586031:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756731.586036:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756731.586040:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756731.586044:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577344:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756731.586050:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756731.586056:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756728/real 1787756731] req@ffff8ffb4790d880 x1874588659577088/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756731.586068:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756731.586070:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756731.586075:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756731.586077:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756731.586079:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586082:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756731.586084:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756731.586089:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659577088/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756731.586099:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586102:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756731.586114:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586116:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756731.586118:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586121:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756731.586139:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756731.586143:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756731.586148:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756731.586152:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756731.586156:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577088:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756731.586165:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756731.586171:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756728/real 1787756731] req@ffff8ffb4790e680 x1874588659576960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756731.586216:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756731.586219:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756731.586225:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756731.586227:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756731.586229:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586232:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756731.586234:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756731.586241:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659576960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756731.586251:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586254:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756731.586258:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586260:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756731.586261:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586263:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756731.586266:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756731.586269:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756731.586273:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756731.586276:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756731.586280:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659576960:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756731.586286:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586289:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756731.586291:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756731.586295:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756731.586302:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756731.586307:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756731.586309:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756731.586315:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756731.586318:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756731.586322:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756731.586324:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586327:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756731.586328:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756731.586330:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756731.586333:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756731.586335:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756731.586337:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756731.586340:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756731.586342:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756731.586345:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756731.586347:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586349:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756731.586351:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756731.586352:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756731.586355:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756731.586356:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756731.586358:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756731.586360:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756731.586362:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756731.586365:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756731.586367:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586369:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756731.586371:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756731.586372:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756731.586375:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756731.586376:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756731.586378:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756731.586381:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756731.586382:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756731.586384:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756731.586386:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586389:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756731.586398:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756731.586401:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756731.586404:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756731.586406:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756732.353462:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756732.353467:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756732.353470:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756733.633339:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756733.633345:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756733.633348:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756733.633351:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756733.633355:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756733.633361:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9788 00000100:00080000:3.0:1787756733.633370:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756733.633390:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756733.633393:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756733.633410:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756733.633414:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756733.633418:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756733.633421:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756733.633423:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756733.633425:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756733.633430:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756733.633433:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756733.633437:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756733.633441:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756733.633445:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633448:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756733.633451:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756733.633461:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756733.633463:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633467:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756733.633470:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756733.633475:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633491:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756733.633508:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659577472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756733.633527:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756733.633535:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633561:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756733.633565:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756733.633569:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756733.633571:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756733.633573:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756733.633577:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756733.633580:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9788 00000100:00000001:2.0:1787756733.633582:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787756733.633587:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756733.633589:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756733.633591:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756733.633593:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756733.633595:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756733.633597:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756733.633599:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756733.633601:0:87148:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:2.0:1787756733.633604:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:3.0:1787756733.633605:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787756733.633608:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787756733.633610:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:2.0:1787756733.633611:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756733.633617:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00100000:2.0:1787756733.633619:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659577472:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756733.633622:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756733.633626:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756733.633627:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756733.633631:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787756733.633634:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787756733.633635:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633639:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756733.633640:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756733.633644:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756733.633646:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756733.633649:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756733.633653:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633655:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756733.633659:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756733.633662:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633664:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756733.633668:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787756733.633678:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787756733.633679:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633682:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:3.0:1787756733.633683:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756733.633687:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756733.633688:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787756733.633690:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633697:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756733.633702:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659577600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756733.633708:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756733.633710:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633713:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756733.633715:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756733.633718:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756733.633720:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756733.633721:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787756733.633726:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756733.633726:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756733.633733:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9788 00000100:00000001:2.0:1787756733.633737:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787756733.633738:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756733.633742:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756733.633743:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756733.633744:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756733.633747:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756733.633748:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756733.633751:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787756733.633752:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:3.0:1787756733.633754:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756733.633757:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756733.633758:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756733.633760:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756733.633761:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756733.633763:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756733.633768:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633770:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304461505 02000000:00000001:3.0:1787756733.633770:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756733.633772:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756733.633776:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633777:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756733.633779:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756733.633781:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633782:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633783:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787756733.633786:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756733.633788:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633790:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787756733.633791:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304461505 02000000:00000001:3.0:1787756733.633792:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756733.633794:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633800:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756733.633801:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756733.633806:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659577728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756733.633807:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756733.633813:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633815:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787756733.633823:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:2.0:1787756733.633824:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756733/real 1787756733] req@ffff8ffb45c30700 x1874588659577472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756733.633826:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633831:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756733.633835:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756733.633842:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756733.633844:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756733.633847:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756733.633850:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000400:00000001:2.0:1787756733.633853:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:3.0:1787756733.633853:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9788 00000400:00000001:2.0:1787756733.633858:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:3.0:1787756733.633858:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756733.633862:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756733.633863:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756733.633865:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000020:00000001:3.0:1787756733.633866:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756733.633868:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787756733.633871:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000020:00000001:3.0:1787756733.633871:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756733.633873:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756733.633876:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756733.633876:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756733.633879:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756733.633882:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787756733.633883:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756733.633887:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756733.633887:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756733.633905:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756733.633908:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756733.633912:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633915:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:2.0:1787756733.633916:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659577472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787756733.633918:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756733.633924:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.633929:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633936:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787756733.633937:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756733.633941:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633942:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756733.633944:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756733.633948:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756733.633949:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787756733.633952:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633954:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756733.633957:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756733.633960:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.633961:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756733.633965:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:3.0:1787756733.633971:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659577856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756733.633972:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756733.633980:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756733.633986:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787756733.633991:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:2.0:1787756733.633994:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659577472:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756733.633996:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756733.634004:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634006:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634010:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756733.634012:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756733.634015:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756733.634019:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756733.634022:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756733.634025:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756733.634028:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756733.634030:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756733.634032:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756733.634034:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634036:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634043:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756733.634046:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634048:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756733.634050:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756733.634051:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756733.634053:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756733.634054:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756733.634056:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756733.634058:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756733.634060:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756733.634063:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577600:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756733.634067:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756733.634068:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756733.634070:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756733.634072:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756733.634074:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634081:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756733.634082:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787756733.634087:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756733.634100:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756733.634105:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756733.634108:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756733.634109:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634111:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756733.634113:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634118:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304460813 00000100:00000001:2.0:1787756733.634120:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304461505 00000100:00000001:2.0:1787756733.634122:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304460813 00000100:00000001:2.0:1787756733.634124:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304461505 00000100:00000001:2.0:1787756733.634126:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756733.634128:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634130:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756733.634134:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756733/real 1787756733] req@ffff8ffb45c33800 x1874588659577600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756733.634143:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756733.634144:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756733.634147:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756733.634149:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756733.634150:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634151:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756733.634153:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756733.634157:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659577600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756733.634163:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634166:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756733.634168:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634170:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756733.634171:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634172:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756733.634216:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756733.634220:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756733.634224:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756733.634227:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756733.634231:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577600:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756733.634236:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756733.634237:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756733.634239:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756733.634242:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756733.634245:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756733.634247:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756733.634252:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756733.634255:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756733.634257:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756733.634259:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756733.634260:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756733.634263:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634269:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756733.634271:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:2.0:1787756733.634275:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756733.634288:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756733.634292:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756733.634295:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756733.634296:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634298:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756733.634301:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634305:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304460811 00000100:00000001:2.0:1787756733.634314:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304461505 00000100:00000001:2.0:1787756733.634317:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304460811 00000100:00000001:2.0:1787756733.634319:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304461505 00000100:00000001:2.0:1787756733.634322:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756733.634324:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634326:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756733.634330:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756733/real 1787756733] req@ffff8ffb45c30e00 x1874588659577728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756733.634338:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756733.634340:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756733.634343:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756733.634344:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756733.634346:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634348:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756733.634350:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756733.634353:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659577728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756733.634359:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634361:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756733.634363:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634365:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756733.634366:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634368:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756733.634387:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756733.634390:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756733.634394:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756733.634397:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756733.634400:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756733.634404:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756733.634405:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756733.634407:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756733.634409:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756733.634411:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756733.634413:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756733.634416:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577856:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756733.634420:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756733.634422:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756733.634424:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756733.634425:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756733.634430:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634435:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756733.634437:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787756733.634440:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756733.634453:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756733.634458:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756733.634461:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756733.634462:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634464:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756733.634467:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634470:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304460811 00000100:00000001:2.0:1787756733.634473:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304461505 00000100:00000001:2.0:1787756733.634475:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304460811 00000100:00000001:2.0:1787756733.634478:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304461505 00000100:00000001:2.0:1787756733.634480:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756733.634482:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634484:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756733.634488:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756733/real 1787756733] req@ffff8ffb45c30000 x1874588659577856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756733.634498:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756733.634500:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756733.634503:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756733.634505:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756733.634506:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634509:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756733.634511:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756733.634515:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659577856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756733.634523:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634525:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756733.634528:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634530:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756733.634531:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634533:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756733.634550:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756733.634552:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756733.634555:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756733.634557:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756733.634560:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659577856:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756733.634564:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634566:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756733.634568:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756733.634569:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756733.634572:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756733.634573:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756733.634575:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756733.634578:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756733.634579:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756733.634581:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756733.634583:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634584:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756733.634586:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756733.634587:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756733.634589:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756733.634590:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756733.634591:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756733.634593:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756733.634594:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756733.634596:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756733.634598:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634600:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756733.634601:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756733.634602:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756733.634604:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756733.634605:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756733.634607:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756733.634608:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756733.634609:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756733.634611:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756733.634612:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634614:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756733.634619:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756733.634621:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756733.634623:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756733.634625:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756735.425433:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756735.425440:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756735.425445:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756736.449490:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756736.449496:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756736.449501:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756738.753356:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756738.753373:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756738.753391:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756738.753397:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756738.753404:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756738.753415:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9793 00000100:00080000:3.0:1787756738.753432:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756738.753437:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756738.753440:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756738.753446:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756738.753451:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756738.753455:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756738.753459:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756738.753462:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756738.753464:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756738.753471:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756738.753474:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756738.753477:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756738.753483:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756738.753488:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753491:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756738.753494:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756738.753515:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756738.753517:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753522:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756738.753528:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756738.753536:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753556:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756738.753571:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659577984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756738.753586:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756738.753592:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753616:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756738.753621:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756738.753626:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756738.753628:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756738.753631:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756738.753635:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756738.753638:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9793 00000100:00080000:3.0:1787756738.753642:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756738.753645:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756738.753647:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756738.753651:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756738.753654:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756738.753656:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756738.753660:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756738.753663:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756738.753665:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756738.753670:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756738.753674:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756738.753677:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756738.753686:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756738.753689:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753693:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756738.753699:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756738.753703:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753708:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.753710:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787756738.753714:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756738.753724:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.753727:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756738.753729:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756738.753734:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756738.753736:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756738.753746:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787756738.753746:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.753758:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753768:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.753769:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756738.753776:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756738.753782:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756738.753793:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659578112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756738.753826:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659577984:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756738.753849:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756738.753857:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756738.753862:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753863:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787756738.753867:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787756738.753875:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.753885:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.753892:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756738.753904:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756738.753915:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756738.753917:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756738.753924:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:3.0:1787756738.753928:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787756738.753932:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787756738.753936:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756738.753947:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756738.753955:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9793 00000100:00080000:3.0:1787756738.753979:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756738.753983:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756738.753990:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:2.0:1787756738.753992:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787756738.754005:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756738.754009:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756738.754014:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756738.754020:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:3.0:1787756738.754022:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756738.754031:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756738.754035:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756738.754037:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756738.754043:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756738.754054:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756738.754056:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756738.754065:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754070:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756738.754072:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787756738.754089:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304466625 02000000:00000001:3.0:1787756738.754089:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756738.754103:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.754107:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756738.754110:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00100000:3.0:1787756738.754113:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756738.754119:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.754124:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754126:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304466625 02000000:00000001:3.0:1787756738.754147:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787756738.754148:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787756738.754155:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754159:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756738.754162:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787756738.754169:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756738.754171:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756738.754231:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.754245:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754254:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756738.754255:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659578240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756738.754277:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756738/real 1787756738] req@ffff8ffb45c33480 x1874588659577984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756738.754278:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756738.754293:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.754305:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756738.754321:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000400:00000001:2.0:1787756738.754326:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:3.0:1787756738.754328:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000400:00000001:2.0:1787756738.754331:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756738.754333:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756738.754342:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756738.754345:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756738.754353:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787756738.754355:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787756738.754366:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9793 00000100:00000001:2.0:1787756738.754369:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.754385:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787756738.754388:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756738.754391:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756738.754395:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756738.754405:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787756738.754409:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659577984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787756738.754412:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756738.754421:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756738.754428:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756738.754445:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756738.754447:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756738.754453:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756738.754458:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756738.754458:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756738.754471:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756738.754475:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756738.754494:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787756738.754495:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.754505:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787756738.754506:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756738.754513:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787756738.754515:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754523:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756738.754525:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756738.754546:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00100000:3.0:1787756738.754548:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756738.754571:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756738.754573:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787756738.754579:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756738.754590:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756738.754597:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1787756738.754598:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756738.754604:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756738.754616:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:2.0:1787756738.754618:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659577984:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787756738.754630:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756738.754639:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787756738.754641:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754650:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756738.754658:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787756738.754658:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.754669:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756738.754677:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00100000:3.0:1787756738.754678:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659578368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000001:2.0:1787756738.754686:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756738.754693:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756738.754702:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787756738.754705:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787756738.754707:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756738.754712:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754714:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756738.754722:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756738.754725:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754730:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.754743:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756738.754746:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754750:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756738.754753:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756738.754756:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756738.754759:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756738.754761:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.754765:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756738.754768:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756738.754772:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756738.754778:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578112:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756738.754783:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756738.754786:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756738.754788:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.754790:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756738.754795:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754825:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756738.754827:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:2.0:1787756738.754834:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756738.754854:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756738.754861:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756738.754864:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756738.754866:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754868:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756738.754872:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754877:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304465934 00000100:00000001:2.0:1787756738.754881:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304466625 00000100:00000001:2.0:1787756738.754885:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304465934 00000100:00000001:2.0:1787756738.754887:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304466625 00000100:00000001:2.0:1787756738.754890:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756738.754893:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.754895:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756738.754900:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756738/real 1787756738] req@ffff8ffb45c31500 x1874588659578112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756738.754911:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756738.754913:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756738.754916:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756738.754918:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756738.754920:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.754922:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756738.754924:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756738.754929:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659578112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756738.754938:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.754941:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756738.754943:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.754945:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756738.754947:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.754949:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756738.754964:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756738.754968:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756738.754971:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756738.754974:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756738.754979:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578112:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756738.754984:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756738.754987:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756738.754989:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.754992:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756738.754994:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756738.754996:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756738.755000:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756738.755004:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756738.755006:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756738.755008:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.755011:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756738.755014:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755019:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756738.755021:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:2.0:1787756738.755026:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756738.755038:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756738.755043:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756738.755045:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756738.755047:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755049:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756738.755053:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755057:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304465931 00000100:00000001:2.0:1787756738.755060:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304466625 00000100:00000001:2.0:1787756738.755063:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304465931 00000100:00000001:2.0:1787756738.755065:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304466625 00000100:00000001:2.0:1787756738.755068:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756738.755070:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755072:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756738.755077:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756738/real 1787756738] req@ffff8ffb45c31180 x1874588659578240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756738.755095:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756738.755096:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756738.755099:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756738.755101:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756738.755103:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755105:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756738.755106:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756738.755111:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659578240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756738.755119:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755121:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756738.755124:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755126:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756738.755128:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755130:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756738.755142:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756738.755145:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756738.755149:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756738.755152:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756738.755156:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756738.755160:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756738.755162:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756738.755164:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.755166:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756738.755168:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756738.755170:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756738.755174:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756738.755177:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756738.755179:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756738.755213:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756738.755216:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756738.755220:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755226:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756738.755228:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787756738.755233:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756738.755246:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756738.755251:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756738.755254:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756738.755256:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755258:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756738.755261:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755266:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304465932 00000100:00000001:2.0:1787756738.755269:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304466625 00000100:00000001:2.0:1787756738.755272:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304465932 00000100:00000001:2.0:1787756738.755275:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304466625 00000100:00000001:2.0:1787756738.755278:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756738.755280:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755283:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756738.755288:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756738/real 1787756738] req@ffff8ffb45c30000 x1874588659578368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756738.755296:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756738.755298:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756738.755301:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756738.755303:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756738.755304:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755306:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756738.755308:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756738.755313:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659578368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756738.755321:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755323:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756738.755327:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755329:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756738.755330:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755332:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756738.755345:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756738.755349:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756738.755352:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756738.755355:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756738.755361:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756738.755368:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755371:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756738.755374:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756738.755377:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756738.755381:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756738.755384:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756738.755386:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756738.755390:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756738.755393:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756738.755396:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756738.755399:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755403:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756738.755405:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756738.755408:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756738.755411:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756738.755413:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756738.755415:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756738.755418:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756738.755420:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756738.755422:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756738.755424:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755427:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756738.755429:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756738.755432:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756738.755434:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756738.755437:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756738.755439:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756738.755441:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756738.755444:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756738.755447:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756738.755450:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755453:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756738.755459:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756738.755463:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756738.755466:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756738.755469:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756739.521530:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756739.521537:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756739.521542:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756740.545495:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756740.545501:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756740.545506:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756743.617484:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756743.617493:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756743.617500:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756743.873817:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756743.873826:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756743.873833:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756743.873839:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756743.873845:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756743.873855:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756743.873862:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756743.873876:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756743.873889:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659578496:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756743.873902:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756743.873911:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756743.873918:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756743.873929:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756743.873938:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.873966:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756743.873973:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:2.0:1787756743.873985:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756743.874044:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756743.874063:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756743.874075:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756743.874083:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874094:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756743.874106:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874120:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304471745 00000100:00000001:2.0:1787756743.874143:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756743.874160:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304471745 00000100:00000001:2.0:1787756743.874176:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756743.874242:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756743.874249:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874257:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756743.874273:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756743/real 1787756743] req@ffff8ffb4790d180 x1874588659578496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756743.874310:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756743.874318:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756743.874328:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756743.874339:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756743.874345:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874353:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756743.874369:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756743.874383:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659578496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756743.874400:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874414:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756743.874426:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874433:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756743.874437:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874444:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756743.874450:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756743.874459:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756743.874473:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756743.874479:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756743.874491:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659578496:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756743.874504:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874510:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756743.874513:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756743.874520:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756743.874526:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756743.874532:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756743.874536:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756743.874545:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756743.874550:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756743.874556:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756743.874560:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874567:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874576:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756743.874582:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874590:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756743.874595:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756743.874600:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756743.874603:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756743.874605:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756743.874609:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756743.874612:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756743.874615:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756743.874621:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578624:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756743.874626:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756743.874631:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756743.874633:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756743.874636:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756743.874641:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874649:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756743.874650:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:2.0:1787756743.874656:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756743.874683:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756743.874689:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756743.874692:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756743.874694:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874696:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756743.874700:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874705:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304471054 00000100:00000001:2.0:1787756743.874709:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304471745 00000100:00000001:2.0:1787756743.874751:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304471054 00000100:00000001:2.0:1787756743.874755:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304471745 00000100:00000001:2.0:1787756743.874758:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756743.874761:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874763:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756743.874770:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756743/real 1787756743] req@ffff8ffb4790fb80 x1874588659578624/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756743.874782:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756743.874783:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756743.874787:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756743.874789:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756743.874791:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874793:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756743.874795:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756743.874800:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659578624/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756743.874809:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874812:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756743.874815:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874817:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756743.874819:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874821:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756743.874835:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756743.874838:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756743.874842:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756743.874845:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756743.874849:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578624:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756743.874857:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756743.874859:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756743.874861:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756743.874864:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756743.874866:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756743.874869:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756743.874873:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756743.874877:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756743.874879:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756743.874881:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756743.874883:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756743.874887:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874894:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756743.874896:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:2.0:1787756743.874901:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756743.874917:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756743.874922:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756743.874925:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756743.874927:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874929:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756743.874933:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874937:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304471051 00000100:00000001:2.0:1787756743.874940:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304471745 00000100:00000001:2.0:1787756743.874943:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304471051 00000100:00000001:2.0:1787756743.874946:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304471745 00000100:00000001:2.0:1787756743.874948:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756743.874951:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.874953:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756743.874957:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756743/real 1787756743] req@ffff8ffb4790ed80 x1874588659578752/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756743.874966:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756743.874968:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756743.874971:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756743.874973:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756743.874975:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874977:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756743.874979:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756743.874984:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659578752/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756743.874992:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874994:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756743.874997:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.874999:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756743.875000:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875002:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756743.875014:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756743.875017:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756743.875020:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756743.875023:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756743.875026:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756743.875030:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756743.875032:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756743.875034:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756743.875036:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756743.875039:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756743.875041:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756743.875045:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578880:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756743.875049:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756743.875051:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756743.875053:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756743.875055:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756743.875058:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.875063:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756743.875065:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:2.0:1787756743.875069:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756743.875081:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756743.875085:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756743.875088:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756743.875089:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.875091:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756743.875094:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.875098:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304471051 00000100:00000001:2.0:1787756743.875101:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304471745 00000100:00000001:2.0:1787756743.875104:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304471051 00000100:00000001:2.0:1787756743.875107:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304471745 00000100:00000001:2.0:1787756743.875112:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756743.875115:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.875119:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756743.875126:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756743/real 1787756743] req@ffff8ffb4790ca80 x1874588659578880/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756743.875142:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756743.875145:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756743.875149:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756743.875152:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756743.875155:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875158:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756743.875161:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756743.875169:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659578880/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756743.875218:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875223:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756743.875228:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875232:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756743.875234:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875238:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756743.875259:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756743.875265:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756743.875272:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756743.875277:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756743.875283:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659578880:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756743.875291:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875295:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756743.875297:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756743.875300:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756743.875303:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756743.875305:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756743.875308:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756743.875311:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756743.875312:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756743.875315:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756743.875317:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875319:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756743.875321:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756743.875323:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756743.875325:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756743.875326:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756743.875327:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756743.875330:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756743.875331:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756743.875333:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756743.875335:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875336:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756743.875338:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756743.875339:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756743.875341:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756743.875342:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756743.875343:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756743.875346:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756743.875347:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756743.875349:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756743.875351:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875353:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756743.875368:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756743.875370:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756743.875373:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756743.875375:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756744.641427:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756744.641433:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756744.641437:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756747.713411:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756747.713417:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756747.713422:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756748.737455:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756748.737462:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756748.737467:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756748.993730:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756748.993744:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756748.993751:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756748.993756:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756748.993762:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756748.993774:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756748.993781:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756748.993788:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756748.993808:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659579008:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756748.993834:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756748.993852:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756748.993861:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756748.993878:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756748.993898:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.993937:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756748.993947:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787756748.993971:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756748.994055:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756748.994078:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756748.994092:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756748.994101:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.994106:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756748.994122:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.994136:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304476865 00000100:00000001:2.0:1787756748.994168:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756748.994271:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304476866 00000100:00000001:2.0:1787756748.994292:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756748.994324:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756748.994340:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.994353:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756748.994386:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756748/real 1787756748] req@ffff8ffb4a534700 x1874588659579008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756748.994457:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756748.994466:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756748.994482:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756748.994491:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756748.994499:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.994506:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756748.994512:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756748.994538:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659579008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756748.994577:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.994592:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756748.994600:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.994608:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756748.994617:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.994648:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756748.994653:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756748.994668:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756748.994684:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756748.994694:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756748.994719:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659579008:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756748.994734:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.994748:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756748.994753:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756748.994759:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756748.994769:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756748.994773:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756748.994777:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756748.994786:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756748.994788:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756748.994792:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756748.994794:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.994798:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.994810:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756748.994813:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.994817:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756748.994820:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756748.994823:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756748.994826:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756748.994830:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756748.994834:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756748.994837:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756748.994840:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756748.994846:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579136:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756748.994852:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756748.994857:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756748.994860:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756748.994864:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756748.994870:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.994883:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756748.994886:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787756748.994894:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756748.994925:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756748.994935:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756748.994940:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756748.994943:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.994946:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756748.994953:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.994960:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304476174 00000100:00000001:2.0:1787756748.994965:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304476865 00000100:00000001:2.0:1787756748.994968:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304476174 00000100:00000001:2.0:1787756748.994971:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304476866 00000100:00000001:2.0:1787756748.994974:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756748.994977:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.994980:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756748.994987:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756748/real 1787756748] req@ffff8ffb4a534a80 x1874588659579136/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756748.995015:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756748.995017:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756748.995021:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756748.995023:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756748.995025:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995028:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756748.995030:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756748.995035:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659579136/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756748.995045:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995049:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756748.995052:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995054:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756748.995055:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995058:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756748.995078:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756748.995082:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756748.995086:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756748.995089:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756748.995093:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579136:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756748.995098:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756748.995101:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756748.995103:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756748.995107:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756748.995110:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756748.995112:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756748.995116:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756748.995121:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756748.995124:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756748.995128:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756748.995131:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756748.995137:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995147:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756748.995150:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787756748.995157:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756748.995224:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756748.995235:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756748.995239:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756748.995243:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995246:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756748.995252:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995259:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304476172 00000100:00000001:2.0:1787756748.995264:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304476865 00000100:00000001:2.0:1787756748.995268:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304476172 00000100:00000001:2.0:1787756748.995273:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304476866 00000100:00000001:2.0:1787756748.995277:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756748.995281:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995285:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756748.995295:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756748/real 1787756748] req@ffff8ffb4a535c00 x1874588659579264/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756748.995314:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756748.995316:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756748.995322:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756748.995326:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756748.995328:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995331:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756748.995335:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756748.995343:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659579264/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756748.995370:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995375:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756748.995380:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995383:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756748.995386:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995391:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756748.995412:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756748.995418:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756748.995423:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756748.995426:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756748.995431:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756748.995437:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756748.995439:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756748.995441:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756748.995444:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756748.995447:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756748.995450:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756748.995454:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579392:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756748.995458:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756748.995461:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756748.995463:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756748.995466:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756748.995470:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995478:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756748.995480:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787756748.995485:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756748.995503:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756748.995510:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756748.995513:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756748.995515:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995517:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756748.995521:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995525:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304476172 00000100:00000001:2.0:1787756748.995529:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304476865 00000100:00000001:2.0:1787756748.995532:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304476172 00000100:00000001:2.0:1787756748.995534:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304476866 00000100:00000001:2.0:1787756748.995537:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756748.995539:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995542:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756748.995546:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756748/real 1787756748] req@ffff8ffb4a537800 x1874588659579392/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756748.995556:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756748.995558:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756748.995560:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756748.995562:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756748.995564:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995567:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756748.995568:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756748.995573:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659579392/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756748.995582:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995585:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756748.995588:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995590:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756748.995592:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995594:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756748.995606:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756748.995609:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756748.995613:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756748.995615:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756748.995646:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579392:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756748.995653:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995656:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756748.995658:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756748.995660:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756748.995663:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756748.995665:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756748.995667:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756748.995672:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756748.995673:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756748.995676:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756748.995677:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995680:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756748.995681:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756748.995683:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756748.995685:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756748.995686:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756748.995688:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756748.995690:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756748.995691:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756748.995693:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756748.995695:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995696:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756748.995698:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756748.995699:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756748.995701:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756748.995702:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756748.995703:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756748.995706:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756748.995707:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756748.995709:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756748.995711:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995713:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756748.995718:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756748.995719:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756748.995722:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756748.995724:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756751.809336:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756751.809340:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756751.809343:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756752.833399:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756752.833405:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756752.833409:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756753.857422:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756753.857433:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756753.857441:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756754.113378:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756754.113390:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756754.113393:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756754.113398:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756754.113404:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756754.113412:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9808 00000100:00080000:3.0:1787756754.113426:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756754.113430:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756754.113433:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756754.113438:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756754.113441:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756754.113445:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756754.113448:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756754.113452:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756754.113454:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756754.113460:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756754.113464:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756754.113468:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756754.113472:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756754.113476:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.113480:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756754.113484:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756754.113499:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756754.113502:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.113507:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756754.113512:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756754.113587:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.113610:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756754.113627:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659579520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756754.113646:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756754.113651:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.113676:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756754.113681:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756754.113685:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756754.113688:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756754.113690:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756754.113694:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756754.113697:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9808 00000100:00080000:3.0:1787756754.113703:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756754.113706:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756754.113708:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756754.113714:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756754.113718:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756754.113721:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756754.113726:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756754.113729:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756754.113732:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756754.113738:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756754.113745:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756754.113747:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756754.113757:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756754.113759:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787756754.113772:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.113775:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756754.113777:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756754.113782:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:3.0:1787756754.113788:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756754.113790:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756754.113794:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787756754.113811:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.113816:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787756754.113816:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.113827:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756754.113830:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756754.113838:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756754.113838:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.113865:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756754.113867:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756754.113877:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756754.113878:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659579520:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756754.113896:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.113897:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756754.113909:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:3.0:1787756754.113912:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659579648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787756754.113916:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.113926:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787756754.113937:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756754.113940:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.113945:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.113949:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756754.113950:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756754.113965:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756754.113968:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756754.113971:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756754.113974:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756754.113979:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9808 00000100:00080000:3.0:1787756754.113983:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756754.113987:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756754.113990:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756754.113993:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756754.113994:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787756754.113997:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:3.0:1787756754.113998:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756754.114010:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000020:00000001:3.0:1787756754.114011:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756754.114025:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756754.114032:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756754.114036:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787756754.114043:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756754.114047:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756754.114049:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756754.114053:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756754.114056:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.114058:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756754.114061:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756754.114065:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.114067:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756754.114072:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756754.114074:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.114076:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756754.114077:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756754.114080:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.114086:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756754.114092:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659579776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756754.114103:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756754.114105:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787756754.114117:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756754.114120:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756754.114127:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756754.114135:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756754.114139:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787756754.114142:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787756754.114143:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756754.114154:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787756754.114158:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787756754.114164:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9808 00000100:00000001:2.0:1787756754.114167:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114174:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787756754.114176:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756754.114246:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756754.114249:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756754.114253:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756754.114254:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756754.114257:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756754.114261:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756754.114264:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756754.114265:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756754.114280:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756754.114286:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756754.114289:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787756754.114291:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756754.114295:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756754.114302:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.114306:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756754.114308:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304481986 00000008:00100000:3.0:1787756754.114310:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756754.114318:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.114323:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114332:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787756754.114333:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756754.114340:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756754.114345:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787756754.114351:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304481986 02000000:00000001:3.0:1787756754.114353:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756754.114366:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114369:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756754.114377:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114379:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756754.114389:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756754.114392:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659579904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756754.114399:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787756754.114415:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756754.114419:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756754.114420:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756754/real 1787756754] req@ffff8ffb45c30e00 x1874588659579520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756754.114430:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787756754.114454:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756754.114457:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756754.114464:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756754.114471:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756754.114474:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114477:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756754.114480:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756754.114487:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659579520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756754.114498:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114503:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756754.114507:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114509:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756754.114511:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114514:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756754.114517:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756754.114522:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756754.114528:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756754.114557:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756754.114566:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659579520:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756754.114576:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114580:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756754.114583:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756754.114588:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756754.114594:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756754.114598:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756754.114601:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756754.114608:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756754.114611:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756754.114615:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756754.114618:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114621:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114632:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756754.114635:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114638:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756754.114641:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756754.114644:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756754.114646:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756754.114648:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.114652:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756754.114655:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756754.114658:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756754.114663:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579648:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756754.114668:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756754.114672:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756754.114674:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.114677:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756754.114681:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114691:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756754.114693:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787756754.114700:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756754.114720:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756754.114727:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756754.114730:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756754.114732:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114734:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756754.114738:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114744:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304481294 00000100:00000001:2.0:1787756754.114748:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304481986 00000100:00000001:2.0:1787756754.114752:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304481294 00000100:00000001:2.0:1787756754.114755:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304481986 00000100:00000001:2.0:1787756754.114758:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756754.114761:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114763:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756754.114768:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756754/real 1787756754] req@ffff8ffb45c33800 x1874588659579648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756754.114781:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756754.114783:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756754.114786:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756754.114789:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756754.114790:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114793:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756754.114795:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756754.114800:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659579648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756754.114809:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114812:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756754.114815:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114817:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756754.114818:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.114820:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756754.114843:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756754.114846:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756754.114850:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756754.114853:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756754.114857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579648:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756754.114863:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756754.114865:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756754.114867:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.114871:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756754.114873:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756754.114875:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756754.114878:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756754.114883:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756754.114885:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756754.114888:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.114890:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756754.114894:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114900:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756754.114902:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787756754.114908:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756754.114920:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756754.114926:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756754.114929:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756754.114932:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114935:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756754.114940:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114947:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304481291 00000100:00000001:2.0:1787756754.114952:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304481986 00000100:00000001:2.0:1787756754.114956:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304481291 00000100:00000001:2.0:1787756754.114960:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304481986 00000100:00000001:2.0:1787756754.114965:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756754.114969:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.114972:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756754.114981:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756754/real 1787756754] req@ffff8ffb45c30700 x1874588659579776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756754.114999:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756754.115001:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756754.115007:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756754.115011:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756754.115014:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115018:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756754.115021:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756754.115030:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659579776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756754.115041:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115044:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756754.115048:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115050:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756754.115052:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115054:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756754.115074:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756754.115077:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756754.115082:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756754.115085:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756754.115089:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756754.115093:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756754.115096:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756754.115098:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.115101:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756754.115104:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756754.115107:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756754.115111:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579904:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756754.115115:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756754.115118:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756754.115120:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756754.115123:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756754.115127:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.115134:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756754.115136:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787756754.115142:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756754.115158:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756754.115164:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756754.115167:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756754.115169:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.115172:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756754.115176:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.115180:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304481291 00000100:00000001:2.0:1787756754.115184:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304481986 00000100:00000001:2.0:1787756754.115233:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304481292 00000100:00000001:2.0:1787756754.115238:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304481986 00000100:00000001:2.0:1787756754.115242:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756754.115247:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.115251:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756754.115261:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756754/real 1787756754] req@ffff8ffb45c32a00 x1874588659579904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756754.115279:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756754.115282:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756754.115288:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756754.115292:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756754.115295:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115299:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756754.115302:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756754.115308:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659579904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756754.115318:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115321:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756754.115324:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115326:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756754.115328:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115330:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756754.115350:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756754.115353:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756754.115356:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756754.115359:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756754.115363:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659579904:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756754.115368:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115371:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756754.115374:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756754.115376:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756754.115380:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756754.115382:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756754.115384:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756754.115387:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756754.115388:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756754.115392:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756754.115394:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115396:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756754.115398:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756754.115400:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756754.115402:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756754.115404:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756754.115405:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756754.115408:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756754.115409:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756754.115411:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756754.115412:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115414:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756754.115416:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756754.115417:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756754.115419:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756754.115420:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756754.115421:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756754.115423:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756754.115424:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756754.115427:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756754.115428:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115430:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756754.115435:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756754.115436:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756754.115439:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756754.115441:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756755.905560:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756755.905567:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756755.905573:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756756.929524:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756756.929532:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756756.929540:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756757.953519:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756757.953527:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756757.953536:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756759.233302:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756759.233316:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756759.233320:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756759.233325:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756759.233332:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756759.233341:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9813 00000100:00080000:3.0:1787756759.233354:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756759.233358:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756759.233361:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756759.233367:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756759.233370:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756759.233374:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756759.233377:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756759.233380:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756759.233382:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756759.233388:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756759.233392:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756759.233395:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756759.233399:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756759.233418:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233421:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756759.233424:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756759.233438:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756759.233463:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233469:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756759.233473:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756759.233479:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233494:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756759.233508:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659580032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756759.233522:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756759.233528:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233550:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756759.233555:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756759.233562:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756759.233565:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756759.233567:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756759.233571:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756759.233575:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9813 00000100:00080000:3.0:1787756759.233580:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756759.233584:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756759.233587:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756759.233592:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756759.233594:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756759.233598:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756759.233601:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756759.233605:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756759.233607:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756759.233611:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756759.233614:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756759.233617:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756759.233625:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756759.233627:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787756759.233629:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233642:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756759.233646:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787756759.233650:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756759.233653:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756759.233659:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:3.0:1787756759.233660:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.233666:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233670:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.233680:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756759.233686:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756759.233687:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756759.233698:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.233699:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233714:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:2.0:1787756759.233724:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659580032:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787756759.233725:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756759.233736:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.233738:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756759.233747:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756759.233752:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233754:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.233758:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756759.233775:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756759.233777:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659580160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756759.233802:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756759.233805:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.233806:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756759.233811:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:3.0:1787756759.233815:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.233818:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787756759.233823:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756759.233830:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756759.233833:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756759.233836:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756759.233842:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756759.233848:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9813 00000100:00080000:3.0:1787756759.233880:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756759.233886:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000800:00000100:2.0:1787756759.233893:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756759.233894:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756759.233906:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756759.233912:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756759.233917:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756759.233918:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756759.233926:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:3.0:1787756759.233928:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756759.233931:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.233941:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756759.233949:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756759.233956:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.233958:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756759.233965:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304487105 00000100:00080000:3.0:1787756759.233975:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756759.233981:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756759.233986:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787756759.233988:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787756759.233994:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756759.234001:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304487105 02000000:00000001:3.0:1787756759.234003:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.234022:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756759.234026:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00100000:3.0:1787756759.234031:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756759.234035:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787756759.234038:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234044:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.234045:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234065:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787756759.234070:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756759.234075:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756759.234083:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756759/real 1787756759] req@ffff8ffb45c31c00 x1874588659580032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756759.234103:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000400:00000001:2.0:1787756759.234108:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787756759.234109:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000400:00000001:2.0:1787756759.234124:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787756759.234127:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234133:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756759.234139:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756759.234141:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234142:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.234161:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756759.234171:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756759.234172:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659580288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756759.234241:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659580032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756759.234241:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756759.234254:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.234261:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756759.234267:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756759.234297:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756759.234307:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787756759.234314:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787756759.234316:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756759.234328:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756759.234328:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787756759.234342:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787756759.234344:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9813 00000100:00000001:2.0:1787756759.234350:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756759.234355:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787756759.234357:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756759.234361:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756759.234364:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756759.234367:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756759.234369:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756759.234383:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00080000:2.0:1787756759.234384:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000020:00000001:3.0:1787756759.234393:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756759.234399:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:2.0:1787756759.234400:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000020:00000001:3.0:1787756759.234417:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756759.234419:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787756759.234436:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:2.0:1787756759.234461:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659580032:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756759.234468:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756759.234476:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787756759.234486:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756759.234487:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756759.234497:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787756759.234498:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.234509:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756759.234509:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756759.234534:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:3.0:1787756759.234534:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787756759.234547:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:3.0:1787756759.234548:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756759.234557:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787756759.234558:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756759.234573:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787756759.234574:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756759.234582:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000008:00000001:3.0:1787756759.234591:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234594:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756759.234597:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234599:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.234606:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787756759.234609:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756759.234612:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.234615:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756759.234617:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756759.234619:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787756759.234622:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234624:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234632:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756759.234636:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756759.234636:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234649:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756759.234653:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756759.234659:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756759.234661:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659580416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787756759.234664:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756759.234672:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756759.234683:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756759.234701:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580160:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756759.234721:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756759.234727:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787756759.234727:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787756759.234736:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.234743:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.234744:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756759.234750:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756759.234758:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234761:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756759.234773:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:2.0:1787756759.234780:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756759.234801:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756759.234808:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756759.234813:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756759.234815:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234817:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756759.234821:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234827:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304486414 00000100:00000001:2.0:1787756759.234832:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304487105 00000100:00000001:2.0:1787756759.234835:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304486414 00000100:00000001:2.0:1787756759.234838:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304487105 00000100:00000001:2.0:1787756759.234841:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756759.234843:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234846:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756759.234851:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756759/real 1787756759] req@ffff8ffb45c33b80 x1874588659580160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756759.234862:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756759.234864:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756759.234867:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756759.234870:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756759.234872:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.234874:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756759.234876:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756759.234881:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659580160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756759.234890:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.234893:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756759.234896:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.234898:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756759.234899:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.234901:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756759.234924:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756759.234927:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756759.234931:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756759.234934:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756759.234938:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580160:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756759.234942:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756759.234945:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756759.234947:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.234950:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756759.234953:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756759.234955:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756759.234959:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756759.234963:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756759.234965:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756759.234967:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.234969:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756759.234973:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.234978:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756759.234980:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:2.0:1787756759.234985:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756759.234998:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756759.235002:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756759.235005:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756759.235007:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235009:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756759.235012:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235016:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304486411 00000100:00000001:2.0:1787756759.235020:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304487105 00000100:00000001:2.0:1787756759.235023:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304486411 00000100:00000001:2.0:1787756759.235025:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304487105 00000100:00000001:2.0:1787756759.235028:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756759.235031:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235033:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756759.235037:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756759/real 1787756759] req@ffff8ffb45c30a80 x1874588659580288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756759.235046:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756759.235048:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756759.235050:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756759.235052:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756759.235053:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235055:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756759.235057:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756759.235061:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659580288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756759.235071:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235074:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756759.235076:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235078:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756759.235080:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235082:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756759.235100:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756759.235103:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756759.235108:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756759.235110:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756759.235115:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756759.235119:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235122:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756759.235123:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756759.235125:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756759.235127:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756759.235129:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756759.235130:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756759.235133:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756759.235134:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756759.235137:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756759.235138:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235140:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756759.235142:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756759.235143:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756759.235146:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756759.235147:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756759.235149:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756759.235151:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756759.235152:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756759.235155:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756759.235156:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235158:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235162:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756759.235164:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235166:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756759.235168:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756759.235169:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756759.235171:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756759.235173:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.235177:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756759.235179:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756759.235181:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756759.235216:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756759.235220:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756759.235223:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756759.235226:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756759.235228:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756759.235232:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235238:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756759.235240:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:2.0:1787756759.235245:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756759.235259:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756759.235264:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756759.235267:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756759.235269:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235271:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756759.235275:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235279:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304486412 00000100:00000001:2.0:1787756759.235282:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304487105 00000100:00000001:2.0:1787756759.235285:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304486412 00000100:00000001:2.0:1787756759.235288:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304487105 00000100:00000001:2.0:1787756759.235291:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756759.235294:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235296:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756759.235301:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756759/real 1787756759] req@ffff8ffb45c31c00 x1874588659580416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756759.235311:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756759.235312:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756759.235315:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756759.235317:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756759.235319:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235321:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756759.235323:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756759.235328:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659580416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756759.235337:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235340:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756759.235343:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235345:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756759.235347:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235349:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756759.235369:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756759.235372:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756759.235376:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756759.235379:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756759.235383:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756759.235387:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235389:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756759.235391:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756759.235392:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756759.235394:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756759.235396:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756759.235398:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756759.235401:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756759.235402:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756759.235405:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756759.235407:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235409:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756759.235413:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756759.235414:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756759.235417:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756759.235420:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756760.001523:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756760.001530:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756760.001536:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756761.025492:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756761.025499:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756761.025504:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756762.049458:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756762.049468:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756762.049474:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756764.097548:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756764.097557:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756764.097564:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756764.353317:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756764.353334:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756764.353338:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756764.353344:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756764.353380:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756764.353392:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9819 00000100:00080000:3.0:1787756764.353410:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756764.353414:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756764.353418:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756764.353424:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756764.353429:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756764.353434:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756764.353437:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756764.353440:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756764.353442:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756764.353448:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756764.353452:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756764.353455:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756764.353462:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756764.353466:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353470:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756764.353473:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756764.353494:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756764.353496:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353501:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756764.353506:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756764.353513:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353532:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756764.353547:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659580544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756764.353563:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756764.353569:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353594:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756764.353599:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756764.353604:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756764.353606:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756764.353608:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756764.353612:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756764.353615:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9819 00000100:00080000:3.0:1787756764.353620:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756764.353622:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756764.353624:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756764.353628:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756764.353630:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756764.353634:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756764.353638:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756764.353641:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756764.353643:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756764.353648:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756764.353651:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756764.353655:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787756764.353657:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787756764.353664:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756764.353668:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787756764.353670:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.353674:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756764.353676:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756764.353681:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:3.0:1787756764.353686:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756764.353687:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787756764.353698:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.353701:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787756764.353702:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.353708:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756764.353711:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756764.353715:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756764.353716:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353723:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756764.353727:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:2.0:1787756764.353730:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659580544:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787756764.353732:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.353744:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787756764.353746:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.353751:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756764.353757:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756764.353759:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659580672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787756764.353765:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787756764.353776:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756764.353780:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.353781:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353785:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756764.353792:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756764.353809:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756764.353810:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756764.353812:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756764.353818:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756764.353818:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787756764.353827:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00080000:3.0:1787756764.353829:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9819 00000100:00080000:3.0:1787756764.353841:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756764.353843:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787756764.353845:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756764.353850:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756764.353854:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756764.353856:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756764.353859:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756764.353862:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756764.353865:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756764.353868:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756764.353872:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756764.353877:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756764.353879:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756764.353883:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756764.353885:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353887:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756764.353890:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756764.353893:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353897:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756764.353903:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756764.353905:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353907:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756764.353909:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756764.353911:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353919:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756764.353924:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659580800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756764.353939:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000800:00000100:2.0:1787756764.353941:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756764.353943:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.353949:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756764.353953:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756764.353958:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756764.353960:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756764.353962:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756764.353966:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756764.353969:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787756764.353977:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787756764.353980:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9819 00000100:00000001:2.0:1787756764.353983:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.353991:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787756764.353997:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756764.354002:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.354005:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756764.354013:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304492225 00000100:00000001:3.0:1787756764.354013:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756764.354025:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756764.354030:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756764.354035:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756764.354036:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787756764.354046:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756764.354051:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304492225 00000100:00000001:3.0:1787756764.354055:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756764.354059:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756764.354062:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756764.354065:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756764.354074:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354080:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:3.0:1787756764.354080:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756764.354088:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756764.354095:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756764/real 1787756764] req@ffff8ffb45c32a00 x1874588659580544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787756764.354095:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756764.354100:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756764.354110:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.354117:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756764.354122:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756764.354131:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.354138:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756764.354149:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756764.354153:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787756764.354155:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787756764.354157:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000400:00000001:2.0:1787756764.354159:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787756764.354161:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756764.354166:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787756764.354168:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354175:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756764.354179:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756764.354225:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756764.354237:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659580928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756764.354239:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756764.354246:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756764.354258:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:2.0:1787756764.354261:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659580544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756764.354264:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756764.354272:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354286:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354292:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756764.354296:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354299:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756764.354301:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354305:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756764.354309:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756764.354314:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756764.354330:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756764.354335:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756764.354345:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659580544:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756764.354381:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354386:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756764.354388:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756764.354392:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756764.354398:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756764.354401:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756764.354405:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756764.354414:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756764.354417:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756764.354421:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756764.354423:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354425:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354437:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756764.354439:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354442:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756764.354445:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756764.354447:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756764.354449:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756764.354451:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.354454:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756764.354457:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756764.354461:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756764.354465:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580672:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756764.354470:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756764.354473:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756764.354476:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.354478:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756764.354482:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354492:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756764.354496:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787756764.354503:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756764.354529:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756764.354536:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756764.354539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756764.354541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354544:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756764.354548:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354554:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304491534 00000100:00000001:2.0:1787756764.354558:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304492225 00000100:00000001:2.0:1787756764.354562:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304491534 00000100:00000001:2.0:1787756764.354565:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304492225 00000100:00000001:2.0:1787756764.354568:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756764.354570:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354572:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756764.354578:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756764/real 1787756764] req@ffff8ffb45c30700 x1874588659580672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756764.354598:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756764.354600:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756764.354603:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756764.354605:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756764.354607:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354609:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756764.354611:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756764.354616:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659580672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756764.354626:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354629:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756764.354632:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354634:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756764.354635:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354637:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756764.354659:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756764.354662:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756764.354666:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756764.354668:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756764.354673:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580672:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756764.354680:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756764.354683:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756764.354686:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.354690:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756764.354695:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756764.354699:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756764.354705:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756764.354712:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756764.354715:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756764.354719:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.354723:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756764.354728:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354739:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756764.354742:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787756764.354750:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756764.354775:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756764.354783:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756764.354788:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756764.354790:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354793:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756764.354798:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354805:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304491531 00000100:00000001:2.0:1787756764.354810:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304492225 00000100:00000001:2.0:1787756764.354815:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304491531 00000100:00000001:2.0:1787756764.354819:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304492225 00000100:00000001:2.0:1787756764.354823:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756764.354828:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.354832:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756764.354840:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756764/real 1787756764] req@ffff8ffb45c33800 x1874588659580800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756764.354856:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756764.354858:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756764.354863:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756764.354866:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756764.354869:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354873:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756764.354875:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756764.354883:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659580800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756764.354896:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354901:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756764.354906:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354909:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756764.354912:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.354915:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756764.354939:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756764.354945:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756764.354952:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756764.354956:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756764.354963:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756764.354970:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756764.354972:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756764.354975:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.354980:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756764.354983:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756764.354988:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756764.354994:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756764.355001:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756764.355005:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756764.355008:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756764.355011:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756764.355017:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.355028:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756764.355031:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:2.0:1787756764.355039:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756764.355063:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756764.355072:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756764.355076:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756764.355079:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.355083:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756764.355088:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.355095:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304491531 00000100:00000001:2.0:1787756764.355100:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304492225 00000100:00000001:2.0:1787756764.355106:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304491531 00000100:00000001:2.0:1787756764.355110:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304492225 00000100:00000001:2.0:1787756764.355115:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756764.355118:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.355120:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756764.355127:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756764/real 1787756764] req@ffff8ffb45c30e00 x1874588659580928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756764.355138:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756764.355140:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756764.355144:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756764.355146:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756764.355148:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355151:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756764.355153:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756764.355158:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659580928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756764.355166:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355169:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756764.355172:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355174:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756764.355175:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355178:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756764.355230:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756764.355235:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756764.355239:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756764.355242:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756764.355246:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659580928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756764.355251:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355253:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756764.355256:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756764.355258:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756764.355262:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756764.355264:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756764.355266:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756764.355269:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756764.355271:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756764.355274:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756764.355276:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355278:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756764.355279:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756764.355282:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756764.355284:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756764.355285:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756764.355287:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756764.355289:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756764.355290:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756764.355292:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756764.355294:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355296:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756764.355297:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756764.355298:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756764.355300:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756764.355301:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756764.355302:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756764.355304:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756764.355305:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756764.355307:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756764.355309:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355311:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756764.355316:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756764.355317:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756764.355320:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756764.355322:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756765.121494:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756765.121501:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756765.121506:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756766.145500:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756766.145506:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756766.145512:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756768.193498:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756768.193505:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756768.193511:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756769.217501:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756769.217508:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756769.217513:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756769.473744:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756769.473753:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756769.473760:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756769.473765:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756769.473770:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756769.473782:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756769.473790:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756769.473796:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756769.473807:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659581056:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756769.473818:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756769.473827:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756769.473832:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756769.473840:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756769.473848:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.473874:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756769.473880:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:2.0:1787756769.473888:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756769.473945:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756769.473965:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756769.473976:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756769.473985:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.473993:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756769.474003:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474012:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304497345 00000100:00000001:2.0:1787756769.474030:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756769.474043:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304497345 00000100:00000001:2.0:1787756769.474053:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756769.474076:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756769.474091:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474103:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756769.474124:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756769/real 1787756769] req@ffff8ffb4790f480 x1874588659581056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756769.474171:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756769.474177:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756769.474250:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756769.474285:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756769.474295:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474303:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756769.474310:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756769.474327:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659581056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756769.474355:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474364:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756769.474380:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474386:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756769.474393:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474402:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756769.474411:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756769.474422:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756769.474439:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756769.474450:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756769.474465:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659581056:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756769.474477:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474485:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756769.474491:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756769.474498:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756769.474509:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756769.474516:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756769.474522:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756769.474531:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756769.474536:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756769.474542:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756769.474545:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474553:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474571:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756769.474576:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474586:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756769.474592:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756769.474596:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756769.474600:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756769.474602:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756769.474605:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756769.474609:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756769.474612:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756769.474619:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659581184:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756769.474625:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756769.474628:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756769.474630:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756769.474633:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756769.474639:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474653:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756769.474655:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133080064 : -123165576471552 : ffff8ffb4790dc00) 00000100:00000001:2.0:1787756769.474662:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756769.474694:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756769.474709:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756769.474714:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756769.474720:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474723:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756769.474733:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474741:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304496654 00000100:00000001:2.0:1787756769.474746:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304497345 00000100:00000001:2.0:1787756769.474751:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304496654 00000100:00000001:2.0:1787756769.474755:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304497345 00000100:00000001:2.0:1787756769.474760:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756769.474762:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474765:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756769.474770:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756769/real 1787756769] req@ffff8ffb4790dc00 x1874588659581184/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756769.474791:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756769.474792:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756769.474795:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756769.474797:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756769.474799:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474803:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756769.474805:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756769.474811:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790dc00 x1874588659581184/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756769.474820:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474823:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756769.474827:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474829:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756769.474830:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.474832:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756769.474856:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756769.474862:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756769.474868:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756769.474874:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756769.474880:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659581184:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756769.474889:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756769.474891:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756769.474894:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756769.474898:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756769.474900:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756769.474903:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756769.474910:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659581312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756769.474915:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756769.474917:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756769.474919:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756769.474921:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756769.474925:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474930:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756769.474932:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133072896 : -123165576478720 : ffff8ffb4790c000) 00000100:00000001:2.0:1787756769.474937:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756769.474951:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756769.474957:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756769.474959:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756769.474961:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474963:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756769.474966:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474970:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304496651 00000100:00000001:2.0:1787756769.474974:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304497345 00000100:00000001:2.0:1787756769.474977:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304496651 00000100:00000001:2.0:1787756769.474979:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304497345 00000100:00000001:2.0:1787756769.474982:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756769.474984:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.474986:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756769.474991:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756769/real 1787756769] req@ffff8ffb4790c000 x1874588659581312/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756769.475000:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756769.475002:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756769.475004:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756769.475006:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756769.475008:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475010:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756769.475012:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756769.475016:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c000 x1874588659581312/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756769.475024:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475027:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756769.475030:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475032:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756769.475034:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475036:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756769.475049:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756769.475053:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756769.475057:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756769.475060:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756769.475064:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659581312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756769.475068:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756769.475069:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756769.475071:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756769.475073:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756769.475075:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756769.475078:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756769.475082:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659581440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756769.475087:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756769.475090:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756769.475093:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756769.475096:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756769.475102:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.475111:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756769.475114:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:2.0:1787756769.475120:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756769.475142:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756769.475151:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756769.475155:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756769.475158:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.475161:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756769.475167:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.475173:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304496651 00000100:00000001:2.0:1787756769.475179:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304497345 00000100:00000001:2.0:1787756769.475227:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304496652 00000100:00000001:2.0:1787756769.475230:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304497345 00000100:00000001:2.0:1787756769.475234:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756769.475238:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.475242:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756769.475250:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756769/real 1787756769] req@ffff8ffb4790ca80 x1874588659581440/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756769.475285:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756769.475288:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756769.475293:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756769.475295:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756769.475297:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475300:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756769.475302:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756769.475307:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659581440/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756769.475317:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475320:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756769.475323:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475325:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756769.475327:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475329:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756769.475348:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756769.475351:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756769.475357:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756769.475360:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756769.475366:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659581440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756769.475372:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475376:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756769.475379:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756769.475382:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756769.475386:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756769.475388:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756769.475390:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756769.475394:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756769.475396:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756769.475399:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756769.475401:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475403:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756769.475404:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756769.475406:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756769.475408:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756769.475410:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756769.475411:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756769.475413:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756769.475414:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756769.475416:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756769.475417:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475419:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756769.475420:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756769.475422:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756769.475423:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756769.475425:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756769.475426:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756769.475428:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756769.475429:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756769.475431:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756769.475432:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475434:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756769.475439:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756769.475440:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756769.475443:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756769.475445:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756770.241541:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756770.241550:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756770.241557:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756772.289558:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756772.289567:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756772.289574:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756773.313504:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756773.313512:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756773.313520:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756774.337479:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756774.337486:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756774.337492:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756774.593360:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756774.593373:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756774.593376:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756774.593381:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756774.593387:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756774.593395:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9829 00000100:00080000:2.0:1787756774.593408:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756774.593413:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756774.593416:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756774.593421:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756774.593424:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756774.593428:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756774.593431:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756774.593434:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756774.593436:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756774.593441:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756774.593445:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756774.593449:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756774.593453:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756774.593457:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593459:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756774.593462:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756774.593477:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756774.593480:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593484:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756774.593487:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756774.593492:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593507:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756774.593520:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659581568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756774.593542:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756774.593551:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593583:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756774.593590:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756774.593597:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756774.593600:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756774.593607:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756774.593614:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756774.593619:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9829 00000100:00080000:2.0:1787756774.593627:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756774.593631:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756774.593634:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756774.593638:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756774.593641:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756774.593644:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756774.593647:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756774.593651:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756774.593654:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756774.593669:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756774.593684:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756774.593698:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756774.593713:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756774.593731:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593740:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756774.593749:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756774.593775:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593790:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756774.593800:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756774.593807:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593823:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756774.593825:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756774.593831:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593848:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756774.593864:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659581696/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756774.593899:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756774.593906:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.593912:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756774.593917:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756774.593922:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756774.593925:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756774.593929:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756774.593934:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756774.593937:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9829 00000100:00080000:2.0:1787756774.593945:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756774.593948:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756774.593950:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756774.593952:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756774.593956:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756774.593959:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756774.593964:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756774.593966:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756774.593968:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756774.593973:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756774.593977:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756774.593980:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756774.593993:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756774.594000:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594005:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756774.594009:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756774.594014:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594018:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756774.594039:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756774.594048:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594060:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756774.594077:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756774.594083:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594095:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756774.594123:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659581824/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756774.594160:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756774.594165:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594270:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756774.594283:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756774.594290:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756774.594298:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756774.594303:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756774.594317:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756774.594332:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9829 00000100:00080000:2.0:1787756774.594351:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756774.594358:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756774.594365:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756774.594370:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756774.594375:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756774.594380:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756774.594391:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756774.594398:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756774.594404:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756774.594422:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756774.594440:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756774.594452:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756774.594466:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756774.594480:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594486:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756774.594493:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756774.594512:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594532:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756774.594556:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756774.594570:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594585:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756774.594598:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756774.594613:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594636:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756774.594664:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659581952/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756774.594712:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756774.594728:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756774.594752:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756776.385460:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756776.385484:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756776.385493:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756777.409487:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756777.409495:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756777.409501:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756778.433541:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756778.433550:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756778.433557:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756779.713341:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756779.713355:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756779.713359:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756779.713364:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756779.713370:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756779.713379:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9834 00000100:00080000:3.0:1787756779.713392:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756779.713397:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756779.713399:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756779.713405:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756779.713408:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756779.713412:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756779.713415:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756779.713418:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756779.713420:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756779.713426:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756779.713429:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756779.713433:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756779.713437:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756779.713441:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713444:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756779.713447:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756779.713463:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756779.713466:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713472:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756779.713477:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756779.713484:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713503:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756779.713520:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659582080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756779.713542:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756779.713551:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713578:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756779.713583:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756779.713591:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756779.713594:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756779.713596:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756779.713600:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756779.713604:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9834 00000100:00080000:3.0:1787756779.713611:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756779.713613:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756779.713616:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756779.713620:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756779.713622:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756779.713626:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756779.713629:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756779.713631:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756779.713633:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756779.713637:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756779.713640:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756779.713643:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756779.713650:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756779.713654:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713661:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756779.713669:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756779.713679:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713686:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756779.713712:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756779.713717:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713726:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756779.713729:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756779.713734:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713743:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756779.713759:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659582208/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756779.713790:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756779.713795:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713799:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756779.713803:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756779.713809:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756779.713811:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756779.713813:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756779.713818:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756779.713826:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9834 00000100:00080000:3.0:1787756779.713831:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756779.713834:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756779.713836:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756779.713839:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756779.713841:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756779.713844:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756779.713848:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756779.713857:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756779.713861:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756779.713871:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756779.713875:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756779.713882:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756779.713895:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756779.713899:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713902:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756779.713905:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756779.713909:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713911:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756779.713918:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756779.713920:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713923:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756779.713925:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756779.713927:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713935:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756779.713942:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659582336/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756779.713962:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756779.713966:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.713970:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756779.713974:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756779.713979:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756779.713985:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756779.713989:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756779.713992:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756779.713998:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9834 00000100:00080000:3.0:1787756779.714005:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756779.714014:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756779.714018:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756779.714025:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756779.714027:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756779.714030:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756779.714035:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756779.714040:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756779.714047:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756779.714053:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756779.714060:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756779.714065:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756779.714068:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756779.714071:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.714073:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756779.714075:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756779.714112:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.714119:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756779.714128:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756779.714134:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.714139:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756779.714146:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756779.714153:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.714174:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756779.714259:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659582464/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756779.714281:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756779.714287:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756779.714294:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756780.481481:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756780.481490:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756780.481496:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756781.505549:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756781.505558:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756781.505564:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756782.529431:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756782.529438:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756782.529443:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756784.577398:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756784.577404:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756784.577409:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756784.833376:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756784.833388:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756784.833391:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756784.833397:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756784.833404:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756784.833415:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9839 00000100:00080000:2.0:1787756784.833433:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756784.833440:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756784.833444:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756784.833452:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756784.833456:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756784.833461:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756784.833467:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756784.833471:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756784.833474:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756784.833484:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756784.833510:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756784.833515:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756784.833520:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756784.833524:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833527:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756784.833531:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756784.833546:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756784.833548:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833553:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756784.833556:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756784.833561:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833575:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756784.833587:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659582592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756784.833602:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756784.833608:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833629:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756784.833634:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756784.833639:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756784.833641:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756784.833643:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756784.833647:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756784.833651:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9839 00000100:00080000:2.0:1787756784.833657:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756784.833660:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756784.833662:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756784.833665:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756784.833668:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756784.833671:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756784.833674:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756784.833677:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756784.833680:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756784.833686:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756784.833690:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756784.833693:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756784.833703:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756784.833713:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833725:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756784.833738:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756784.833752:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833764:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756784.833778:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756784.833788:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833799:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756784.833805:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756784.833813:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833831:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756784.833842:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659582720/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756784.833870:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756784.833874:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.833880:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756784.833884:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756784.833888:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756784.833891:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756784.833894:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756784.833898:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756784.833902:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9839 00000100:00080000:2.0:1787756784.833909:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756784.833911:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756784.833915:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756784.833925:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756784.833933:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756784.833938:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756784.833946:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756784.833956:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756784.833964:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756784.833979:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756784.834011:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756784.834021:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756784.834031:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756784.834038:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834042:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756784.834046:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756784.834055:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834059:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756784.834067:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756784.834072:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834076:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756784.834079:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756784.834085:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834096:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756784.834104:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659582848/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756784.834144:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756784.834153:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834162:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756784.834173:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756784.834231:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756784.834233:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756784.834237:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756784.834244:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756784.834256:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9839 00000100:00080000:2.0:1787756784.834267:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756784.834277:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756784.834286:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756784.834293:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756784.834300:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756784.834306:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756784.834312:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756784.834321:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756784.834325:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756784.834339:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756784.834346:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756784.834352:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756784.834358:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756784.834367:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834376:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756784.834382:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756784.834389:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834396:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756784.834411:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756784.834420:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834433:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756784.834440:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756784.834448:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834464:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756784.834488:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659582976/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756784.834519:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756784.834528:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756784.834537:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756785.601468:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756785.601476:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756785.601488:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756786.625438:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756786.625446:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756786.625453:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756788.673500:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756788.673508:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756788.673515:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756789.697484:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756789.697493:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756789.697499:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756789.953381:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756789.953455:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756789.953458:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756789.953463:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756789.953469:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756789.953479:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9844 00000100:00080000:3.0:1787756789.953493:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756789.953497:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756789.953500:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756789.953505:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756789.953508:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756789.953512:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756789.953516:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756789.953519:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756789.953521:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756789.953527:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756789.953531:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756789.953535:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756789.953539:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756789.953543:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953546:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756789.953549:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756789.953576:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756789.953579:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953582:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756789.953586:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756789.953592:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953609:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756789.953621:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659583104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756789.953635:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756789.953641:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953659:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756789.953663:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756789.953668:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756789.953670:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756789.953673:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756789.953676:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756789.953679:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9844 00000100:00080000:3.0:1787756789.953684:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756789.953686:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756789.953688:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756789.953690:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756789.953692:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756789.953695:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756789.953698:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756789.953701:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756789.953703:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756789.953707:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756789.953710:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756789.953712:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756789.953718:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756789.953721:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953723:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756789.953729:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756789.953734:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953737:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756789.953742:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756789.953745:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953748:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756789.953750:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756789.953753:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953759:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756789.953767:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659583232/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756789.953783:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756789.953789:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.953798:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756789.953805:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756789.953812:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756789.953818:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756789.953823:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756789.953830:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756789.953841:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9844 00000100:00080000:3.0:1787756789.953850:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756789.953858:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756789.953865:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756789.953870:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756789.953875:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756789.953882:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756789.953889:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756789.953948:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756789.953955:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756789.953973:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756789.953985:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756789.953990:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756789.953997:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756789.954002:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954006:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756789.954011:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756789.954018:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954022:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756789.954033:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756789.954038:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954043:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756789.954049:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756789.954054:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954064:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756789.954076:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659583360/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756789.954096:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756789.954102:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954110:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756789.954116:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756789.954129:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756789.954132:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756789.954135:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756789.954147:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756789.954151:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9844 00000100:00080000:3.0:1787756789.954160:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756789.954166:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756789.954170:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756789.954222:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756789.954225:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756789.954228:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756789.954231:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756789.954234:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756789.954237:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756789.954242:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756789.954246:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756789.954251:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756789.954259:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756789.954272:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954285:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756789.954294:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756789.954304:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954316:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756789.954332:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756789.954343:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954353:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756789.954361:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756789.954371:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954389:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756789.954409:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659583488/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756789.954444:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756789.954454:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756789.954465:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756790.721589:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756790.721597:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756790.721604:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756792.769468:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756792.769477:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756792.769484:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756793.793460:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756793.793469:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756793.793474:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756794.817571:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756794.817578:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756794.817583:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756795.073313:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756795.073332:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756795.073335:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756795.073340:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756795.073346:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756795.073355:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9849 00000100:00080000:3.0:1787756795.073370:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756795.073374:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756795.073377:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756795.073382:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756795.073385:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756795.073389:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756795.073392:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756795.073394:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756795.073396:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756795.073401:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756795.073404:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756795.073407:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756795.073411:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756795.073414:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073417:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756795.073419:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756795.073448:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756795.073450:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073455:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756795.073458:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756795.073465:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073482:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756795.073495:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659583616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756795.073515:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756795.073520:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073540:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756795.073543:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756795.073547:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756795.073549:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756795.073551:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756795.073554:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756795.073556:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9849 00000100:00080000:3.0:1787756795.073560:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756795.073561:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756795.073563:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756795.073565:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756795.073566:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756795.073568:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756795.073571:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756795.073573:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756795.073574:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756795.073577:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756795.073579:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756795.073580:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756795.073586:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756795.073588:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073607:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756795.073612:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756795.073616:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073618:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756795.073623:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756795.073625:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073629:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756795.073632:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756795.073635:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073644:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756795.073655:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659583744/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756795.073671:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756795.073685:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073697:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756795.073708:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756795.073721:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756795.073726:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756795.073732:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756795.073742:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756795.073751:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9849 00000100:00080000:3.0:1787756795.073765:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756795.073770:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756795.073773:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756795.073781:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756795.073785:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756795.073790:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756795.073793:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756795.073799:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756795.073801:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756795.073806:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756795.073833:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756795.073852:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756795.073856:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756795.073858:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073860:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756795.073863:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756795.073871:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073872:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756795.073877:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756795.073879:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073882:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756795.073884:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756795.073886:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073894:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756795.073902:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659583872/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756795.073926:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756795.073930:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.073935:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756795.073940:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756795.073945:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756795.073947:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756795.073950:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756795.073956:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756795.073960:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9849 00000100:00080000:3.0:1787756795.073967:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756795.073970:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756795.073973:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756795.073977:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756795.073980:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756795.073982:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756795.073986:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756795.073989:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756795.073991:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756795.073998:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756795.074007:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756795.074016:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756795.074021:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756795.074024:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.074027:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756795.074031:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756795.074037:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.074040:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756795.074047:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756795.074051:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.074056:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756795.074057:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756795.074060:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.074067:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756795.074078:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659584000/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756795.074094:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756795.074101:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756795.074107:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756796.865491:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756796.865500:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756796.865507:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756797.889480:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756797.889492:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756797.889500:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756798.913564:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756798.913572:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756798.913578:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756800.193340:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756800.193351:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756800.193354:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756800.193358:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756800.193363:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756800.193370:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9854 00000100:00080000:3.0:1787756800.193382:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756800.193386:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756800.193389:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756800.193393:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756800.193396:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756800.193400:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756800.193403:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756800.193406:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756800.193407:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756800.193413:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756800.193416:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756800.193419:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756800.193422:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756800.193425:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193428:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756800.193430:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756800.193441:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756800.193442:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193445:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756800.193448:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756800.193452:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193464:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756800.193474:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659584128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756800.193485:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756800.193489:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193508:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756800.193511:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756800.193515:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756800.193516:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756800.193518:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756800.193521:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756800.193523:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9854 00000100:00080000:3.0:1787756800.193527:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756800.193529:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756800.193531:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756800.193533:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756800.193535:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756800.193538:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756800.193541:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756800.193543:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756800.193546:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756800.193550:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756800.193553:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756800.193555:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756800.193561:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756800.193576:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193581:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756800.193590:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756800.193599:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193608:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756800.193613:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756800.193620:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193622:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756800.193623:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756800.193628:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193636:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756800.193650:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659584256/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756800.193681:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756800.193693:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193701:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756800.193709:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756800.193743:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756800.193744:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756800.193746:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756800.193756:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756800.193767:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9854 00000100:00080000:3.0:1787756800.193775:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756800.193783:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756800.193784:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756800.193790:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756800.193794:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756800.193798:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756800.193802:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756800.193805:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756800.193807:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756800.193828:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756800.193837:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756800.193845:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756800.193852:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756800.193855:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193859:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756800.193863:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756800.193867:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193870:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756800.193878:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756800.193880:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193882:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756800.193886:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756800.193891:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193901:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756800.193910:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659584384/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756800.193938:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756800.193947:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.193952:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756800.193956:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756800.193963:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756800.193971:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756800.193974:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756800.193990:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756800.194004:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9854 00000100:00080000:3.0:1787756800.194018:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756800.194028:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756800.194031:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756800.194044:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756800.194046:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756800.194048:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756800.194052:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756800.194057:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756800.194063:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756800.194085:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756800.194098:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756800.194107:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756800.194120:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756800.194136:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.194150:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756800.194162:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756800.194168:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.194176:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756800.194257:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756800.194259:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.194262:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756800.194264:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756800.194266:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.194275:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756800.194287:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659584512/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756800.194362:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756800.194371:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756800.194377:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756800.961442:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756800.961448:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756800.961454:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756801.985462:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756801.985467:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756801.985472:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756803.009538:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756803.009545:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756803.009550:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756805.057521:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756805.057528:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756805.057533:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756805.313322:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756805.313336:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756805.313339:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756805.313344:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756805.313350:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756805.313361:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9860 00000100:00080000:3.0:1787756805.313376:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756805.313381:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756805.313385:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756805.313391:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756805.313408:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756805.313412:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756805.313416:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756805.313419:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756805.313421:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756805.313427:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756805.313435:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756805.313440:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756805.313447:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756805.313452:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.313457:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756805.313462:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756805.313479:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756805.313482:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.313488:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756805.313492:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756805.313500:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.313517:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756805.313533:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659584640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756805.313549:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756805.313555:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.313589:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756805.313593:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756805.313598:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756805.313600:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756805.313603:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756805.313607:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756805.313611:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9860 00000100:00080000:3.0:1787756805.313616:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756805.313619:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756805.313621:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756805.313624:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756805.313655:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756805.313659:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756805.313662:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756805.313665:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756805.313667:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756805.313672:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756805.313678:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756805.313682:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756805.313690:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756805.313694:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.313699:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756805.313708:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756805.313716:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.313722:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756805.313734:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756805.313739:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.313741:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756805.313743:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756805.313756:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756805.313760:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787756805.313763:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.313767:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756805.313774:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756805.313776:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.313778:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756805.313790:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659584768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787756805.313792:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756805.313804:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756805.313820:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787756805.313822:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.313828:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.313843:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756805.313845:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659584640:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787756805.313864:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787756805.313867:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756805.313878:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756805.313882:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756805.313887:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756805.313896:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.313924:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756805.313927:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787756805.313933:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787756805.313985:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756805.313987:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756805.313990:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756805.313995:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756805.314001:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9860 00000800:00000100:2.0:1787756805.314008:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787756805.314015:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756805.314028:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756805.314036:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756805.314039:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.314042:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756805.314051:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.314057:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304533185 00000100:00000001:2.0:1787756805.314073:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756805.314081:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304533185 00000100:00000001:2.0:1787756805.314087:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756805.314091:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756805.314096:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.314099:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756805.314111:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756805/real 1787756805] req@ffff8ffb45c32a00 x1874588659584640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756805.314129:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756805.314132:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756805.314136:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756805.314139:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756805.314142:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.314145:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756805.314149:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756805.314156:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659584640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756805.314167:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.314173:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756805.314177:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.314179:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756805.314181:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.314230:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756805.314235:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756805.314240:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756805.314249:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756805.314257:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756805.314259:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787756805.314262:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756805.314272:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:2.0:1787756805.314273:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659584640:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756805.314288:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.314292:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756805.314295:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756805.314300:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756805.314306:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756805.314310:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756805.314314:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756805.314322:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756805.314325:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756805.314332:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756805.314335:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.314338:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.314350:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756805.314353:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.314358:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756805.314364:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756805.314367:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756805.314369:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756805.314371:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756805.314375:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756805.314379:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787756805.314384:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787756805.314385:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756805.314396:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00100000:2.0:1787756805.314402:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659584768:192.168.201.118@tcp:38:kworker.0 00000020:00000001:3.0:1787756805.314408:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756805.314424:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787756805.314426:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756805.314437:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787756805.314439:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756805.314447:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756805.314456:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787756805.314457:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756805.314468:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756805.314469:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756805.314486:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787756805.314492:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787756805.314495:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756805.314498:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 02000000:00000001:3.0:1787756805.314505:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.314511:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756805.314512:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00100000:3.0:1787756805.314521:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756805.314526:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.314529:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756805.314538:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756805.314547:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787756805.314548:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756805.314561:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756805.314566:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787756805.314568:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787756805.314570:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.314576:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756805.314582:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.314583:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.314589:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756805.314597:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756805.314603:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659584896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756805.314607:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304532494 00000100:00000001:2.0:1787756805.314621:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304533185 00000100:00000001:2.0:1787756805.314661:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304532494 00000100:00000001:2.0:1787756805.314677:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304533185 00000100:00000001:3.0:1787756805.314690:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756805.314691:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756805.314705:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.314706:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.314726:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787756805.314733:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756805.314743:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756805.314744:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756805/real 1787756805] req@ffff8ffb45c30700 x1874588659584768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756805.314763:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756805.314774:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756805.314781:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756805.314793:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756805.314807:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9860 00000100:00080000:3.0:1787756805.314824:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000400:00000001:2.0:1787756805.314833:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787756805.314834:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:2.0:1787756805.314841:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756805.314843:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756805.314854:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787756805.314855:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000020:00000001:3.0:1787756805.314869:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787756805.314872:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000020:00000001:3.0:1787756805.314879:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756805.314881:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.314895:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:3.0:1787756805.314895:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756805.314912:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756805.314914:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756805.314925:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787756805.314936:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659584768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756805.314941:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756805.314959:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756805.314974:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787756805.314985:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787756805.314986:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756805.315002:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787756805.315002:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.315023:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756805.315024:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315035:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00100000:3.0:1787756805.315035:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756805.315047:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787756805.315050:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315052:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756805.315061:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756805.315070:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756805.315072:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.315076:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756805.315079:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756805.315083:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.315093:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315105:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:3.0:1787756805.315108:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659585024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756805.315115:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756805.315141:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756805.315152:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756805.315154:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787756805.315160:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756805.315171:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756805.315172:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659584768:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756805.315248:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315251:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756805.315255:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756805.315261:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756805.315268:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756805.315272:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756805.315275:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756805.315281:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756805.315283:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756805.315287:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756805.315290:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315294:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315301:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756805.315303:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315307:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756805.315309:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756805.315312:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756805.315316:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756805.315319:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756805.315323:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756805.315326:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756805.315329:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756805.315336:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659584896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756805.315341:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756805.315345:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756805.315348:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756805.315351:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756805.315355:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315362:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756805.315365:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787756805.315370:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756805.315387:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756805.315393:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756805.315396:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756805.315398:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315400:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756805.315403:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315408:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304532492 00000100:00000001:2.0:1787756805.315411:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304533185 00000100:00000001:2.0:1787756805.315415:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304532492 00000100:00000001:2.0:1787756805.315417:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304533185 00000100:00000001:2.0:1787756805.315420:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756805.315422:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315424:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756805.315430:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756805/real 1787756805] req@ffff8ffb45c33800 x1874588659584896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756805.315449:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756805.315451:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756805.315454:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756805.315456:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756805.315458:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315460:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756805.315462:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756805.315466:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659584896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756805.315475:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315479:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756805.315481:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315483:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756805.315485:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315487:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756805.315505:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756805.315508:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756805.315512:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756805.315515:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756805.315518:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659584896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756805.315523:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756805.315525:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756805.315527:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756805.315530:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756805.315532:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756805.315535:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756805.315539:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756805.315542:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756805.315544:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756805.315547:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756805.315549:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756805.315553:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315558:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756805.315560:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787756805.315565:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756805.315577:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756805.315581:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756805.315584:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756805.315586:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315588:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756805.315591:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315595:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304532492 00000100:00000001:2.0:1787756805.315598:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304533185 00000100:00000001:2.0:1787756805.315601:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304532492 00000100:00000001:2.0:1787756805.315604:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304533185 00000100:00000001:2.0:1787756805.315607:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756805.315609:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315611:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756805.315616:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756805/real 1787756805] req@ffff8ffb45c30000 x1874588659585024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756805.315625:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756805.315649:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756805.315652:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756805.315655:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756805.315657:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315659:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756805.315661:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756805.315666:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659585024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756805.315675:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315678:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756805.315682:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315685:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756805.315686:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315688:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756805.315707:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756805.315711:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756805.315715:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756805.315718:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756805.315722:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756805.315726:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315728:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756805.315730:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756805.315732:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756805.315734:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756805.315736:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756805.315737:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756805.315740:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756805.315742:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756805.315745:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756805.315747:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315749:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756805.315751:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756805.315752:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756805.315754:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756805.315756:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756805.315757:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756805.315759:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756805.315761:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756805.315762:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756805.315764:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315766:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756805.315770:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756805.315771:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756805.315773:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756805.315776:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756806.081424:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756806.081435:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756806.081446:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756807.105370:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756807.105375:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756807.105379:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756809.153490:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756809.153499:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756809.153507:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756810.177446:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756810.177453:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756810.177458:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756810.433259:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756810.433272:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756810.433275:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756810.433280:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756810.433285:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756810.433294:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9865 00000100:00080000:3.0:1787756810.433306:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756810.433310:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756810.433313:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756810.433319:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756810.433322:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756810.433326:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756810.433330:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756810.433332:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756810.433334:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756810.433340:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756810.433344:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433357:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756810.433361:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756810.433365:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433368:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756810.433371:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433387:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756810.433390:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433394:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756810.433397:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756810.433402:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433417:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756810.433427:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659585152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756810.433437:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756810.433441:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433457:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756810.433459:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756810.433470:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756810.433472:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756810.433474:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756810.433478:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756810.433480:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9865 00000100:00080000:3.0:1787756810.433485:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756810.433488:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756810.433490:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756810.433492:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756810.433494:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756810.433496:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756810.433500:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756810.433502:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756810.433503:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756810.433508:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756810.433511:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433514:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756810.433520:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756810.433522:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433523:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756810.433526:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787756810.433529:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756810.433532:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:3.0:1787756810.433533:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433553:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756810.433556:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756810.433559:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433559:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433567:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756810.433568:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756810.433571:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756810.433573:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756810.433574:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756810.433577:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756810.433578:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433580:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756810.433586:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659585152:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756810.433587:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433594:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:3.0:1787756810.433596:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659585280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787756810.433601:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756810.433606:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433606:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756810.433611:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756810.433613:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787756810.433615:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756810.433619:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787756810.433620:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756810.433626:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756810.433628:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756810.433629:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756810.433632:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756810.433635:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9865 00000100:00080000:3.0:1787756810.433638:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756810.433641:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756810.433642:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756810.433643:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756810.433647:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000020:00000001:3.0:1787756810.433648:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756810.433651:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787756810.433652:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787756810.433655:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756810.433660:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756810.433663:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756810.433664:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756810.433669:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756810.433672:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433675:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756810.433678:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756810.433681:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433683:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756810.433687:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756810.433690:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433692:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433697:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756810.433699:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787756810.433701:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756810.433702:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756810.433705:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756810.433707:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433712:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756810.433717:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659585408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756810.433718:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756810.433723:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756810.433725:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756810.433727:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433728:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433733:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756810.433734:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756810.433738:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756810.433740:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756810.433741:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433742:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756810.433745:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756810.433748:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304538305 00000100:00000001:3.0:1787756810.433748:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756810.433753:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9865 00000100:00080000:3.0:1787756810.433756:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756810.433758:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756810.433760:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756810.433762:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787756810.433763:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787756810.433763:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756810.433767:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756810.433770:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756810.433772:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304538305 00000100:00000001:3.0:1787756810.433773:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756810.433776:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756810.433781:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787756810.433781:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756810.433783:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433786:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756810.433787:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756810.433790:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756810.433791:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433793:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433795:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787756810.433796:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756810.433800:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756810.433803:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756810.433804:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756810/real 1787756810] req@ffff8ffb45c31180 x1874588659585152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756810.433805:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756810.433812:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756810.433815:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756810.433817:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000400:00000001:2.0:1787756810.433819:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787756810.433819:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756810.433821:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787756810.433823:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756810.433826:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433827:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756810.433832:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756810.433835:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756810.433836:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659585536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756810.433839:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756810.433843:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756810.433848:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756810.433852:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756810.433853:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659585152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756810.433855:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433863:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.433867:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756810.433869:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.433871:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756810.433872:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.433875:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756810.433876:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756810.433879:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756810.433882:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756810.433883:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756810.433887:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659585152:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756810.433891:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.433893:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756810.433895:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756810.433897:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756810.433901:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756810.433904:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756810.433906:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756810.433911:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756810.433913:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756810.433915:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756810.433916:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.433918:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.433925:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756810.433927:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433929:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756810.433931:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756810.433932:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756810.433934:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756810.433936:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756810.433938:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756810.433940:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756810.433943:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756810.433946:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585280:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756810.433950:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756810.433952:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756810.433954:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756810.433956:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756810.433959:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433965:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756810.433966:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:2.0:1787756810.433972:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756810.433985:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756810.433990:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756810.433993:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756810.433994:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.433996:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756810.433999:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434004:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304537613 00000100:00000001:2.0:1787756810.434006:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304538305 00000100:00000001:2.0:1787756810.434008:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304537613 00000100:00000001:2.0:1787756810.434010:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304538305 00000100:00000001:2.0:1787756810.434012:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756810.434014:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434015:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756810.434019:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756810/real 1787756810] req@ffff8ffb45c31500 x1874588659585280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756810.434026:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756810.434027:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756810.434029:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756810.434030:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756810.434031:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434032:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756810.434033:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756810.434036:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659585280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756810.434042:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434043:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756810.434046:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434047:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756810.434047:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434048:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756810.434066:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756810.434068:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756810.434070:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756810.434071:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756810.434074:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585280:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756810.434077:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756810.434078:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756810.434079:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756810.434080:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756810.434081:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756810.434083:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756810.434085:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756810.434087:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756810.434089:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756810.434090:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756810.434091:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756810.434094:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434098:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756810.434099:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:2.0:1787756810.434102:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756810.434110:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756810.434112:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756810.434114:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756810.434115:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434116:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756810.434118:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434120:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304537610 00000100:00000001:2.0:1787756810.434122:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304538305 00000100:00000001:2.0:1787756810.434124:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304537610 00000100:00000001:2.0:1787756810.434126:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304538305 00000100:00000001:2.0:1787756810.434127:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756810.434129:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434130:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756810.434132:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756810/real 1787756810] req@ffff8ffb45c33480 x1874588659585408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756810.434137:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756810.434138:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756810.434139:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756810.434140:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756810.434140:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434141:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756810.434142:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756810.434144:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659585408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756810.434154:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434155:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756810.434157:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434158:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756810.434159:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434160:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756810.434174:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756810.434175:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756810.434177:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756810.434178:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756810.434180:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756810.434206:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756810.434208:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756810.434209:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756810.434210:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756810.434212:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756810.434213:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756810.434216:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756810.434218:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756810.434219:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756810.434221:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756810.434222:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756810.434224:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434228:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756810.434229:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787756810.434232:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756810.434239:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756810.434243:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756810.434244:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756810.434245:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434247:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756810.434249:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434251:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304537611 00000100:00000001:2.0:1787756810.434253:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304538305 00000100:00000001:2.0:1787756810.434255:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304537611 00000100:00000001:2.0:1787756810.434257:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304538305 00000100:00000001:2.0:1787756810.434258:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756810.434259:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434261:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756810.434263:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756810/real 1787756810] req@ffff8ffb45c30000 x1874588659585536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756810.434268:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756810.434268:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756810.434270:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756810.434271:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756810.434272:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434273:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756810.434274:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756810.434277:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659585536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756810.434281:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434282:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756810.434284:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434285:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756810.434285:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434286:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756810.434301:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756810.434303:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756810.434305:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756810.434307:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756810.434309:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756810.434312:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434313:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756810.434314:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756810.434315:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756810.434316:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756810.434317:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756810.434318:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756810.434320:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756810.434321:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756810.434322:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756810.434323:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434324:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756810.434325:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756810.434326:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756810.434327:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756810.434328:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756810.434329:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756810.434330:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756810.434331:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756810.434332:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756810.434333:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434333:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756810.434334:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756810.434335:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756810.434336:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756810.434336:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756810.434337:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756810.434338:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756810.434338:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756810.434339:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756810.434340:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434341:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756810.434344:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756810.434344:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756810.434346:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756810.434347:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756811.201422:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756811.201433:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756811.201439:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756813.249479:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756813.249498:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756813.249504:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756814.273584:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756814.273592:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756814.273597:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756815.297405:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756815.297411:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756815.297414:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756815.553340:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756815.553352:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756815.553354:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756815.553359:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756815.553363:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756815.553383:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9870 00000100:00080000:3.0:1787756815.553395:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756815.553398:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756815.553400:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756815.553405:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756815.553408:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756815.553412:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756815.553415:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756815.553417:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756815.553418:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756815.553423:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756815.553425:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756815.553428:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756815.553432:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756815.553499:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553501:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756815.553504:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756815.553515:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756815.553516:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553519:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756815.553521:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756815.553525:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553554:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756815.553565:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659585664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756815.553577:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756815.553582:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553600:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756815.553603:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756815.553607:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756815.553608:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756815.553610:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756815.553614:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756815.553616:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9870 00000100:00080000:3.0:1787756815.553620:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756815.553622:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756815.553623:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756815.553626:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756815.553628:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756815.553631:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756815.553633:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756815.553635:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756815.553636:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756815.553640:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756815.553643:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756815.553645:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756815.553650:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756815.553654:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553655:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756815.553661:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756815.553667:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553672:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756815.553678:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756815.553682:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553685:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756815.553687:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756815.553691:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.553696:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756815.553700:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.553709:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756815.553716:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:3.0:1787756815.553718:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659585792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756815.553721:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756815.553727:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756815.553740:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756815.553748:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756815.553756:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553756:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756815.553787:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756815.553788:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659585664:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756815.553801:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.553803:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787756815.553811:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787756815.553814:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787756815.553817:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:2.0:1787756815.553820:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553822:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:2.0:1787756815.553826:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787756815.553826:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756815.553836:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756815.553839:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9870 02000000:00000001:2.0:1787756815.553843:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756815.553845:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756815.553852:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756815.553854:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756815.553859:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756815.553861:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756815.553864:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756815.553878:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756815.553880:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756815.553882:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756815.553887:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756815.553890:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756815.553892:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756815.553897:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756815.553900:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787756815.553901:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.553905:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:3.0:1787756815.553909:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756815.553913:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00100000:3.0:1787756815.553917:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756815.553922:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553924:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756815.553929:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756815.553930:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553931:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756815.553942:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756815.553945:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553950:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756815.553956:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659585920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756815.553964:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756815.553965:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.553967:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756815.553969:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756815.553972:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756815.553973:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756815.553976:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756815.553979:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756815.553981:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9870 00000100:00080000:3.0:1787756815.553985:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756815.553987:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756815.553988:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756815.553990:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756815.553995:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756815.554000:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000800:00000100:2.0:1787756815.554001:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787756815.554014:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756815.554019:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756815.554022:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756815.554026:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756815.554029:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554030:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787756815.554034:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756815.554038:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756815.554039:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787756815.554042:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554044:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.554046:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756815.554052:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:3.0:1787756815.554054:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756815.554060:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554066:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.554068:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756815.554076:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756815.554078:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.554079:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756815.554079:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756815.554081:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756815.554091:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554092:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304543425 00000100:00100000:3.0:1787756815.554100:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659586048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756815.554106:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756815.554108:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554109:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756815.554112:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554122:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304543425 00000100:00000001:2.0:1787756815.554130:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756815.554141:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756815.554145:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554149:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756815.554160:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756815/real 1787756815] req@ffff8ffb45c33800 x1874588659585664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756815.554175:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756815.554177:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756815.554182:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756815.554237:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756815.554240:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554243:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756815.554246:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756815.554252:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659585664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756815.554261:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554266:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756815.554269:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554271:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756815.554273:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554276:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756815.554278:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756815.554282:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756815.554287:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756815.554290:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756815.554295:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659585664:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756815.554301:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554304:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756815.554306:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756815.554309:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756815.554314:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756815.554318:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756815.554320:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756815.554327:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756815.554329:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756815.554333:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756815.554335:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554338:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554347:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756815.554350:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554353:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756815.554355:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756815.554357:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756815.554359:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756815.554361:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756815.554364:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756815.554366:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756815.554369:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756815.554373:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585792:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756815.554377:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756815.554379:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756815.554381:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756815.554383:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756815.554386:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554393:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756815.554394:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787756815.554400:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756815.554414:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756815.554420:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756815.554422:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756815.554424:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554425:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756815.554429:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554434:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304542734 00000100:00000001:2.0:1787756815.554437:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304543425 00000100:00000001:2.0:1787756815.554440:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304542734 00000100:00000001:2.0:1787756815.554443:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304543425 00000100:00000001:2.0:1787756815.554467:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756815.554473:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554476:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756815.554482:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756815/real 1787756815] req@ffff8ffb45c30700 x1874588659585792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756815.554505:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756815.554506:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756815.554510:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756815.554512:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756815.554514:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554516:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756815.554518:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756815.554522:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659585792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756815.554530:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554533:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756815.554536:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554538:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756815.554539:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554541:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756815.554574:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756815.554579:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756815.554584:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756815.554587:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756815.554593:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585792:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756815.554600:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756815.554603:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756815.554606:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756815.554610:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756815.554613:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756815.554617:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756815.554622:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756815.554629:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756815.554632:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756815.554635:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756815.554639:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756815.554644:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554654:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756815.554657:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787756815.554663:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756815.554679:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756815.554685:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756815.554688:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756815.554689:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554691:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756815.554694:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554699:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304542731 00000100:00000001:2.0:1787756815.554702:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304543425 00000100:00000001:2.0:1787756815.554704:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304542731 00000100:00000001:2.0:1787756815.554707:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304543425 00000100:00000001:2.0:1787756815.554709:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756815.554711:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554713:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756815.554718:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756815/real 1787756815] req@ffff8ffb45c32a00 x1874588659585920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756815.554726:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756815.554727:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756815.554732:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756815.554734:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756815.554736:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554738:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756815.554740:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756815.554744:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659585920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756815.554758:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554761:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756815.554764:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554766:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756815.554767:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554769:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756815.554786:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756815.554788:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756815.554801:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756815.554803:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756815.554806:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659585920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756815.554811:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756815.554813:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756815.554814:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756815.554817:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756815.554820:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756815.554822:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756815.554825:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756815.554828:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756815.554831:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756815.554833:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756815.554834:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756815.554838:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554843:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756815.554844:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:2.0:1787756815.554849:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756815.554858:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756815.554863:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756815.554865:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756815.554867:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554869:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756815.554872:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554875:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304542731 00000100:00000001:2.0:1787756815.554878:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304543425 00000100:00000001:2.0:1787756815.554881:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304542731 00000100:00000001:2.0:1787756815.554883:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304543425 00000100:00000001:2.0:1787756815.554886:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756815.554888:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.554889:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756815.554893:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756815/real 1787756815] req@ffff8ffb45c30e00 x1874588659586048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756815.554901:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756815.554902:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756815.554904:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756815.554905:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756815.554907:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554908:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756815.554910:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756815.554914:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659586048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756815.554922:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554924:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756815.554926:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554928:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756815.554929:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554931:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756815.554949:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756815.554952:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756815.554955:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756815.554957:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756815.554970:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756815.554974:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554976:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756815.554978:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756815.554980:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756815.554982:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756815.554984:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756815.554985:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756815.554987:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756815.554989:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756815.554991:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756815.554992:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.554994:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756815.554995:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756815.554997:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756815.554999:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756815.555000:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756815.555001:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756815.555003:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756815.555004:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756815.555006:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756815.555007:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.555009:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756815.555010:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756815.555011:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756815.555012:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756815.555013:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756815.555014:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756815.555016:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756815.555017:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756815.555019:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756815.555020:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.555022:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756815.555026:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756815.555027:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756815.555029:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756815.555032:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756817.345514:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756817.345523:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756817.345530:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756818.369564:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756818.369571:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756818.369576:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756819.393443:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756819.393448:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756819.393452:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756820.673672:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756820.673679:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756820.673684:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756820.673688:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756820.673691:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756820.673697:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756820.673703:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756820.673709:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756820.673718:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659586176:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756820.673730:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756820.673737:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756820.673741:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756820.673746:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756820.673753:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.673773:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756820.673778:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787756820.673784:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756820.673829:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756820.673848:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756820.673855:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756820.673860:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.673865:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756820.673872:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.673883:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304548545 00000100:00000001:2.0:1787756820.673898:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756820.673910:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304548545 00000100:00000001:2.0:1787756820.673920:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756820.673938:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756820.673946:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.673952:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756820.673963:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756820/real 1787756820] req@ffff8ffb4a537b80 x1874588659586176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756820.673993:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756820.673998:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756820.674003:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756820.674009:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756820.674014:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674022:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756820.674028:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756820.674037:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659586176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756820.674059:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674065:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756820.674075:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674079:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756820.674085:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674092:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756820.674099:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756820.674106:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756820.674114:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756820.674122:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756820.674130:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659586176:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756820.674135:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674137:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756820.674139:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756820.674145:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756820.674152:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756820.674157:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756820.674161:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756820.674167:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756820.674173:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756820.674177:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756820.674182:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674215:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674224:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756820.674226:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674229:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756820.674231:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756820.674237:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756820.674241:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756820.674245:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756820.674248:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756820.674253:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756820.674259:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756820.674267:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586304:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756820.674275:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756820.674279:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756820.674283:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756820.674287:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756820.674292:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674303:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756820.674307:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787756820.674316:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756820.674346:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756820.674352:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756820.674375:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756820.674378:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674382:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756820.674387:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674394:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304547854 00000100:00000001:2.0:1787756820.674398:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304548545 00000100:00000001:2.0:1787756820.674402:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304547854 00000100:00000001:2.0:1787756820.674406:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304548545 00000100:00000001:2.0:1787756820.674409:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756820.674413:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674416:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756820.674424:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756820/real 1787756820] req@ffff8ffb4a536300 x1874588659586304/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756820.674440:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756820.674442:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756820.674446:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756820.674449:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756820.674451:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674454:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756820.674456:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756820.674461:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659586304/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756820.674469:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674471:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756820.674474:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674475:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756820.674477:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674478:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756820.674492:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756820.674495:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756820.674498:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756820.674500:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756820.674503:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586304:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756820.674507:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756820.674509:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756820.674511:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756820.674513:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756820.674515:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756820.674517:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756820.674520:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756820.674524:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756820.674526:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756820.674528:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756820.674530:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756820.674533:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674540:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756820.674542:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787756820.674546:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756820.674559:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756820.674564:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756820.674567:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756820.674568:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674570:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756820.674573:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674576:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304547851 00000100:00000001:2.0:1787756820.674578:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304548545 00000100:00000001:2.0:1787756820.674581:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304547851 00000100:00000001:2.0:1787756820.674583:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304548545 00000100:00000001:2.0:1787756820.674585:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756820.674587:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674589:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756820.674593:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756820/real 1787756820] req@ffff8ffb4a537800 x1874588659586432/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756820.674600:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756820.674602:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756820.674604:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756820.674606:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756820.674607:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674609:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756820.674610:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756820.674614:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659586432/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756820.674620:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674622:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756820.674625:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674627:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756820.674628:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674630:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756820.674641:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756820.674644:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756820.674647:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756820.674649:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756820.674652:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756820.674655:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674657:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756820.674658:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756820.674660:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756820.674662:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756820.674664:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756820.674665:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756820.674667:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756820.674668:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756820.674671:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756820.674672:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674674:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756820.674675:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756820.674676:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756820.674678:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756820.674679:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756820.674680:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756820.674681:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756820.674682:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756820.674684:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756820.674685:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674687:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674690:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756820.674691:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674693:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756820.674695:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756820.674696:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756820.674697:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756820.674699:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756820.674701:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756820.674703:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756820.674705:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756820.674708:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756820.674711:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756820.674712:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756820.674713:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756820.674715:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756820.674717:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674721:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756820.674723:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787756820.674726:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756820.674736:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756820.674739:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756820.674741:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756820.674743:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674744:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756820.674747:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674750:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304547851 00000100:00000001:2.0:1787756820.674753:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304548545 00000100:00000001:2.0:1787756820.674755:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304547851 00000100:00000001:2.0:1787756820.674757:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304548545 00000100:00000001:2.0:1787756820.674759:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756820.674761:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674763:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756820.674767:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756820/real 1787756820] req@ffff8ffb4a535c00 x1874588659586560/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756820.674774:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756820.674775:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756820.674777:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756820.674778:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756820.674780:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674782:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756820.674783:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756820.674787:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659586560/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756820.674794:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674796:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756820.674798:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674800:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756820.674801:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674802:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756820.674813:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756820.674816:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756820.674818:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756820.674820:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756820.674823:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756820.674826:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674828:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756820.674829:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756820.674831:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756820.674832:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756820.674834:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756820.674835:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756820.674838:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756820.674839:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756820.674841:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756820.674842:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674844:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756820.674846:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756820.674848:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756820.674850:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756820.674852:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756821.441496:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756821.441502:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756821.441507:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756822.465472:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756822.465480:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756822.465485:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756823.489471:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756823.489479:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756823.489485:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756825.537524:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756825.537534:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756825.537540:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756825.793409:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756825.793425:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756825.793429:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756825.793434:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756825.793441:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756825.793452:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9880 00000100:00080000:3.0:1787756825.793469:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756825.793474:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756825.793477:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756825.793483:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756825.793488:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756825.793492:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756825.793495:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756825.793498:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756825.793500:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756825.793507:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756825.793510:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756825.793525:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756825.793530:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756825.793535:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.793537:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756825.793541:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756825.793558:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756825.793561:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.793566:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756825.793570:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756825.793577:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.793596:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756825.793611:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659586688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756825.793627:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756825.793633:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.793659:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756825.793664:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756825.793670:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756825.793672:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756825.793674:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756825.793678:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756825.793681:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9880 00000100:00080000:3.0:1787756825.793685:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756825.793687:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756825.793689:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756825.793692:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756825.793694:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756825.793697:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756825.793700:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756825.793702:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756825.793704:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756825.793707:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756825.793711:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756825.793713:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756825.793722:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756825.793724:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.793735:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756825.793742:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756825.793746:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.793749:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756825.793754:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756825.793758:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.793760:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756825.793763:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756825.793766:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.793771:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756825.793779:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659586816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756825.793791:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756825.793806:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756825.793809:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756825.793810:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.793821:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756825.793822:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.793834:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787756825.793834:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787756825.793849:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756825.793852:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756825.793859:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756825.793867:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787756825.793868:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756825.793881:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756825.793882:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:2.0:1787756825.793896:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756825.793904:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9880 00000100:00100000:2.0:1787756825.793917:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659586688:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787756825.793922:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756825.793926:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787756825.793932:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787756825.793935:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787756825.793940:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787756825.793941:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756825.793949:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787756825.793959:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756825.793964:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787756825.793967:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756825.793970:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756825.793977:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756825.793983:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756825.793985:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787756825.793993:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787756825.793993:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756825.794010:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756825.794012:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787756825.794028:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 02000000:00000001:3.0:1787756825.794030:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756825.794050:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794051:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787756825.794061:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756825.794067:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756825.794071:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.794073:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756825.794080:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756825.794082:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.794086:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756825.794088:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756825.794093:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.794102:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756825.794110:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659586944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756825.794120:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756825.794150:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756825.794155:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.794161:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794166:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787756825.794167:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756825.794175:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756825.794179:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787756825.794180:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756825.794186:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794189:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756825.794201:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794208:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304553666 00000100:00080000:3.0:1787756825.794234:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756825.794240:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756825.794246:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9880 00000100:00080000:3.0:1787756825.794253:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756825.794256:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756825.794258:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756825.794262:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756825.794263:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756825.794290:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756825.794295:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756825.794299:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756825.794302:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756825.794308:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756825.794313:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794321:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787756825.794321:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756825.794338:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756825.794341:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304553666 02000000:00000001:3.0:1787756825.794348:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.794353:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756825.794355:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00100000:3.0:1787756825.794357:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787756825.794363:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787756825.794366:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794371:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.794372:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794380:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787756825.794386:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756825.794390:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756825.794396:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756825/real 1787756825] req@ffff8ffb45c33b80 x1874588659586688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756825.794397:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756825.794404:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756825.794410:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.794429:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787756825.794432:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756825.794442:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:3.0:1787756825.794450:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659587072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756825.794451:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756825.794458:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756825.794465:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794481:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756825.794494:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756825.794495:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756825.794522:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756825.794533:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756825.794543:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659586688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756825.794555:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794560:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756825.794565:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794568:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756825.794571:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794576:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756825.794579:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756825.794584:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756825.794589:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756825.794593:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756825.794597:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659586688:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756825.794603:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794606:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756825.794608:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756825.794612:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756825.794617:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756825.794621:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756825.794624:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756825.794629:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756825.794632:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756825.794635:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756825.794637:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794640:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794649:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756825.794652:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794655:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756825.794657:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756825.794660:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756825.794662:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756825.794665:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756825.794668:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756825.794671:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756825.794674:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756825.794679:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586816:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756825.794683:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756825.794687:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756825.794689:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756825.794691:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756825.794695:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794703:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756825.794705:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:2.0:1787756825.794711:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756825.794746:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756825.794752:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756825.794755:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756825.794756:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794758:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756825.794761:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794766:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304552974 00000100:00000001:2.0:1787756825.794769:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304553666 00000100:00000001:2.0:1787756825.794772:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304552974 00000100:00000001:2.0:1787756825.794775:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304553666 00000100:00000001:2.0:1787756825.794777:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756825.794780:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794782:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756825.794787:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756825/real 1787756825] req@ffff8ffb45c30a80 x1874588659586816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756825.794797:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756825.794798:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756825.794801:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756825.794803:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756825.794804:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794806:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756825.794808:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756825.794812:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659586816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756825.794820:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794822:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756825.794824:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794826:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756825.794827:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794829:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756825.794844:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756825.794846:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756825.794849:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756825.794852:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756825.794855:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586816:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756825.794859:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756825.794861:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756825.794863:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756825.794866:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756825.794868:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756825.794870:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756825.794873:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756825.794877:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756825.794879:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756825.794881:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756825.794883:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756825.794886:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794891:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756825.794893:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:2.0:1787756825.794897:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756825.794907:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756825.794911:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756825.794914:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756825.794915:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794917:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756825.794919:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794923:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304552971 00000100:00000001:2.0:1787756825.794926:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304553666 00000100:00000001:2.0:1787756825.794928:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304552971 00000100:00000001:2.0:1787756825.794931:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304553666 00000100:00000001:2.0:1787756825.794933:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756825.794935:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.794937:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756825.794941:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756825/real 1787756825] req@ffff8ffb45c31c00 x1874588659586944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756825.794949:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756825.794950:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756825.794952:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756825.794954:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756825.794955:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794957:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756825.794958:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756825.794962:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659586944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756825.794976:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794978:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756825.794981:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794982:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756825.794984:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.794985:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756825.795018:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756825.795021:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756825.795025:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756825.795028:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756825.795031:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659586944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756825.795036:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756825.795038:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756825.795040:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756825.795043:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756825.795045:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756825.795048:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756825.795051:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659587072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756825.795055:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756825.795057:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756825.795059:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756825.795061:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756825.795064:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.795069:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756825.795071:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:2.0:1787756825.795075:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756825.795088:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756825.795092:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756825.795094:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756825.795096:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.795099:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756825.795101:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.795105:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304552971 00000100:00000001:2.0:1787756825.795108:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304553666 00000100:00000001:2.0:1787756825.795111:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304552971 00000100:00000001:2.0:1787756825.795114:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304553666 00000100:00000001:2.0:1787756825.795116:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756825.795119:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.795121:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756825.795125:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756825/real 1787756825] req@ffff8ffb45c30e00 x1874588659587072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756825.795155:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756825.795156:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756825.795158:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756825.795159:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756825.795161:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795163:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756825.795164:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756825.795168:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659587072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756825.795175:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795178:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756825.795180:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795182:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756825.795183:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795185:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756825.795197:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756825.795199:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756825.795202:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756825.795204:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756825.795208:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659587072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756825.795213:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795215:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756825.795216:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756825.795218:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756825.795220:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756825.795221:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756825.795222:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756825.795225:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756825.795226:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756825.795228:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756825.795230:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795231:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756825.795233:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756825.795234:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756825.795236:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756825.795237:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756825.795238:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756825.795240:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756825.795241:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756825.795243:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756825.795244:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795246:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756825.795247:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756825.795248:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756825.795249:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756825.795251:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756825.795252:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756825.795303:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756825.795305:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756825.795307:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756825.795308:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795310:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756825.795314:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756825.795316:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756825.795318:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756825.795319:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756826.561500:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756826.561508:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756826.561515:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756827.585461:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756827.585469:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756827.585475:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756829.633430:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756829.633437:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756829.633442:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756830.657378:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756830.657381:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756830.657384:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756830.913762:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756830.913777:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756830.913791:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756830.913798:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756830.913812:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756830.913857:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756830.913869:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756830.913882:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756830.913909:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659587200:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756830.913931:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756830.913940:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756830.913947:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756830.913955:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756830.913999:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914020:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756830.914025:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787756830.914041:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756830.914121:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756830.914167:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756830.914281:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756830.914288:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914302:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756830.914321:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914330:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304558786 00000100:00000001:2.0:1787756830.914345:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756830.914357:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304558786 00000100:00000001:2.0:1787756830.914368:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756830.914388:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756830.914399:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914402:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756830.914416:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756830/real 1787756830] req@ffff8ffb4a534a80 x1874588659587200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756830.914453:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756830.914458:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756830.914470:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756830.914474:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756830.914478:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914481:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756830.914486:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756830.914494:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659587200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756830.914503:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914506:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756830.914509:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914511:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756830.914512:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914514:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756830.914516:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756830.914519:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756830.914523:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756830.914526:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756830.914529:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659587200:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756830.914534:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756830.914535:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756830.914537:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756830.914539:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756830.914542:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756830.914544:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756830.914547:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659587328:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756830.914550:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756830.914553:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756830.914555:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756830.914556:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756830.914576:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914584:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756830.914586:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787756830.914591:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756830.914609:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756830.914614:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756830.914617:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756830.914619:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914621:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756830.914624:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914628:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304558094 00000100:00000001:2.0:1787756830.914631:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304558786 00000100:00000001:2.0:1787756830.914634:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304558094 00000100:00000001:2.0:1787756830.914637:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304558786 00000100:00000001:2.0:1787756830.914639:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756830.914642:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914644:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756830.914648:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756830/real 1787756830] req@ffff8ffb4a534700 x1874588659587328/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756830.914657:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756830.914658:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756830.914661:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756830.914662:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756830.914664:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914666:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756830.914667:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756830.914671:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659587328/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756830.914686:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914689:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756830.914695:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914698:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756830.914699:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914703:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756830.914726:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756830.914742:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756830.914749:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756830.914753:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756830.914758:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659587328:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756830.914772:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914776:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756830.914779:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756830.914782:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756830.914788:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756830.914791:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756830.914794:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756830.914801:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756830.914804:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756830.914809:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756830.914811:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914814:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756830.914816:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756830.914819:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756830.914822:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756830.914824:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756830.914826:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756830.914830:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756830.914832:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756830.914835:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756830.914837:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914841:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.914851:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756830.914855:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914859:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756830.914861:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756830.914864:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756830.914867:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756830.914869:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756830.914873:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756830.914877:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756830.914881:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756830.914888:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659587456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756830.914895:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756830.914899:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756830.914901:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756830.914905:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756830.914910:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914921:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756830.914924:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787756830.914930:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756830.914952:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756830.914958:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756830.914961:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756830.914963:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914964:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756830.914968:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.914988:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304558091 00000100:00000001:2.0:1787756830.914990:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304558786 00000100:00000001:2.0:1787756830.914993:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304558091 00000100:00000001:2.0:1787756830.914995:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304558786 00000100:00000001:2.0:1787756830.914997:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756830.914999:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.915000:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756830.915004:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756830/real 1787756830] req@ffff8ffb4a535c00 x1874588659587456/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756830.915018:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756830.915019:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756830.915022:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756830.915023:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756830.915024:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915026:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756830.915027:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756830.915031:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659587456/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756830.915037:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915039:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756830.915041:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915059:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756830.915060:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915062:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756830.915074:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756830.915077:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756830.915080:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756830.915082:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756830.915085:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659587456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756830.915090:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756830.915091:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756830.915093:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756830.915096:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756830.915098:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756830.915100:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756830.915103:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659587584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756830.915107:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756830.915109:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756830.915112:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756830.915114:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756830.915117:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.915122:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756830.915124:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787756830.915128:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756830.915140:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756830.915144:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756830.915146:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756830.915148:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.915150:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756830.915152:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.915156:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304558091 00000100:00000001:2.0:1787756830.915159:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304558786 00000100:00000001:2.0:1787756830.915161:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304558091 00000100:00000001:2.0:1787756830.915164:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304558786 00000100:00000001:2.0:1787756830.915166:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756830.915168:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.915170:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756830.915237:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756830/real 1787756830] req@ffff8ffb4a537800 x1874588659587584/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756830.915245:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756830.915247:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756830.915249:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756830.915251:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756830.915252:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915254:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756830.915255:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756830.915260:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659587584/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756830.915266:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915268:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756830.915271:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915273:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756830.915274:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915275:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756830.915287:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756830.915289:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756830.915292:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756830.915294:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756830.915297:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659587584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756830.915300:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915302:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756830.915303:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756830.915304:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756830.915306:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756830.915308:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756830.915309:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756830.915311:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756830.915312:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756830.915314:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756830.915316:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915317:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756830.915318:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756830.915319:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756830.915321:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756830.915321:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756830.915322:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756830.915324:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756830.915325:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756830.915326:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756830.915327:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915329:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756830.915332:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756830.915333:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756830.915335:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756830.915337:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756831.681449:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756831.681455:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756831.681460:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756833.729389:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756833.729395:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756833.729399:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756834.753656:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756834.753665:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756834.753670:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756835.777466:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756835.777478:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756835.777486:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756836.033353:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756836.033367:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756836.033370:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756836.033375:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756836.033381:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756836.033393:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9890 00000100:00080000:2.0:1787756836.033410:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756836.033416:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756836.033420:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756836.033427:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756836.033432:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756836.033438:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756836.033443:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756836.033448:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756836.033451:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756836.033460:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756836.033466:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756836.033472:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756836.033478:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756836.033484:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033488:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756836.033493:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756836.033510:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756836.033512:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033517:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756836.033520:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756836.033525:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033540:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756836.033553:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659587712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756836.033568:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756836.033574:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033604:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756836.033608:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756836.033613:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756836.033615:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756836.033617:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756836.033621:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756836.033624:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9890 00000100:00080000:2.0:1787756836.033630:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756836.033633:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756836.033635:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756836.033640:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756836.033643:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756836.033647:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756836.033650:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756836.033652:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756836.033655:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756836.033659:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756836.033662:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756836.033667:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756836.033687:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756836.033695:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033715:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756836.033731:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756836.033747:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033759:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756836.033777:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756836.033785:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033794:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756836.033804:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756836.033820:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033839:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756836.033856:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659587840/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756836.033870:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756836.033887:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.033909:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756836.033920:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756836.033928:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756836.033932:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756836.033937:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756836.033944:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756836.033952:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9890 00000100:00080000:2.0:1787756836.033963:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756836.033969:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756836.033973:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756836.033987:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756836.033994:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756836.034016:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756836.034025:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756836.034029:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756836.034036:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756836.034060:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756836.034069:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756836.034074:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756836.034120:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756836.034127:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034136:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756836.034146:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756836.034156:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034167:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756836.034181:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756836.034262:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034277:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756836.034285:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756836.034295:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034309:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756836.034333:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659587968/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756836.034358:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756836.034364:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034382:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756836.034397:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756836.034426:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756836.034435:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756836.034444:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756836.034462:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756836.034485:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9890 00000100:00080000:2.0:1787756836.034512:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756836.034520:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756836.034531:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756836.034543:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756836.034550:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756836.034558:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756836.034578:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756836.034593:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756836.034605:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756836.034626:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756836.034633:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756836.034651:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756836.034666:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756836.034677:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034687:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756836.034701:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756836.034709:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034715:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756836.034737:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756836.034741:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034746:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756836.034749:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756836.034754:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034766:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756836.034791:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659588096/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756836.034833:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756836.034850:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756836.034859:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756837.825484:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756837.825489:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756837.825492:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756838.849354:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756838.849358:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756838.849362:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756839.873559:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756839.873568:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756839.873574:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756841.153389:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756841.153401:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756841.153405:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756841.153409:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756841.153415:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756841.153426:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9895 00000100:00080000:2.0:1787756841.153440:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756841.153443:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756841.153445:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756841.153450:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756841.153455:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756841.153459:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756841.153462:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756841.153465:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756841.153468:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756841.153475:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756841.153478:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756841.153483:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756841.153488:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756841.153492:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153495:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756841.153499:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756841.153512:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756841.153514:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153517:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756841.153520:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756841.153525:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153538:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756841.153550:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659588224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756841.153562:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756841.153567:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153588:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756841.153592:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756841.153599:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756841.153601:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756841.153603:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756841.153606:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756841.153609:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9895 00000100:00080000:2.0:1787756841.153615:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756841.153619:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756841.153621:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756841.153626:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756841.153630:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756841.153635:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756841.153639:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756841.153644:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756841.153653:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787756841.153653:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756841.153664:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756841.153673:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:2.0:1787756841.153674:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756841.153679:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756841.153689:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153693:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.153703:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756841.153704:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756841.153709:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756841.153717:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756841.153718:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756841.153724:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153733:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:3.0:1787756841.153734:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659588224:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1787756841.153744:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787756841.153746:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:2.0:1787756841.153755:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.153759:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787756841.153761:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.153771:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756841.153775:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756841.153779:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756841.153781:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153791:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756841.153795:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756841.153799:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153808:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756841.153816:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659588352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756841.153834:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756841.153855:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153866:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756841.153871:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756841.153875:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756841.153878:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756841.153881:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756841.153884:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756841.153888:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9895 00000100:00080000:2.0:1787756841.153894:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756841.153897:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756841.153900:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756841.153902:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756841.153905:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756841.153909:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:3.0:1787756841.153910:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756841.153919:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756841.153925:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756841.153928:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756841.153932:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756841.153937:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756841.153940:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756841.153944:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756841.153946:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153948:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756841.153952:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756841.153955:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153957:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756841.153961:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756841.153963:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153966:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756841.153967:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756841.153969:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.153977:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756841.153982:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659588480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756841.153992:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756841.154039:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.154043:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756841.154048:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756841.154053:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756841.154055:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756841.154058:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756841.154063:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756841.154068:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9895 00000100:00080000:2.0:1787756841.154073:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756841.154076:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756841.154078:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756841.154082:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756841.154084:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756841.154086:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756841.154090:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756841.154093:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756841.154095:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756841.154100:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756841.154104:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756841.154107:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756841.154112:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756841.154116:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.154120:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756841.154123:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756841.154128:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.154130:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756841.154138:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756841.154140:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.154143:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756841.154146:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756841.154149:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.154157:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756841.154165:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659588608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756841.154177:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756841.154230:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756841.154242:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154344:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756841.154351:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:3.0:1787756841.154361:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756841.154462:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756841.154481:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756841.154490:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756841.154495:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154498:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756841.154508:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154515:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304569026 00000100:00000001:3.0:1787756841.154529:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756841.154536:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304569026 00000100:00000001:3.0:1787756841.154541:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756841.154546:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756841.154550:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154554:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756841.154564:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756841/real 1787756841] req@ffff8ffb47a38380 x1874588659588224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756841.154579:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756841.154582:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756841.154588:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756841.154591:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756841.154593:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154597:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756841.154600:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756841.154605:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659588224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756841.154616:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154620:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756841.154625:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154627:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756841.154628:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154632:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756841.154635:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756841.154639:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756841.154646:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756841.154649:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756841.154655:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659588224:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756841.154662:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154666:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756841.154668:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756841.154672:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756841.154677:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756841.154681:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756841.154684:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756841.154691:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756841.154693:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756841.154698:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756841.154699:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154703:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154713:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756841.154716:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154719:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756841.154722:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756841.154724:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756841.154726:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756841.154728:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.154731:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756841.154734:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756841.154737:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756841.154742:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588352:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756841.154746:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756841.154749:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756841.154751:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.154754:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756841.154757:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154764:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756841.154766:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:3.0:1787756841.154773:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756841.154789:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756841.154794:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756841.154797:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756841.154799:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154801:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756841.154805:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154810:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304568334 00000100:00000001:3.0:1787756841.154814:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304569026 00000100:00000001:3.0:1787756841.154818:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304568334 00000100:00000001:3.0:1787756841.154821:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304569026 00000100:00000001:3.0:1787756841.154824:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756841.154826:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154829:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756841.154834:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756841/real 1787756841] req@ffff8ffb47a38000 x1874588659588352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756841.154843:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756841.154845:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756841.154855:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756841.154857:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756841.154859:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154862:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756841.154863:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756841.154868:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659588352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756841.154876:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154879:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756841.154881:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154883:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756841.154885:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.154887:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756841.154910:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756841.154913:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756841.154917:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756841.154921:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756841.154925:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588352:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756841.154930:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756841.154932:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756841.154934:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.154938:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756841.154940:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756841.154942:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756841.154946:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756841.154950:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756841.154952:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756841.154954:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.154956:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756841.154959:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.154965:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756841.154967:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:3.0:1787756841.154972:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756841.154984:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756841.154989:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756841.154992:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756841.155025:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.155029:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756841.155046:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.155053:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304568331 00000100:00000001:3.0:1787756841.155058:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304569026 00000100:00000001:3.0:1787756841.155063:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304568331 00000100:00000001:3.0:1787756841.155068:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304569026 00000100:00000001:3.0:1787756841.155071:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756841.155075:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.155079:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756841.155088:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756841/real 1787756841] req@ffff8ffb47a3b480 x1874588659588480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756841.155102:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756841.155104:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756841.155108:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756841.155111:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756841.155113:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155116:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756841.155118:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756841.155123:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659588480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756841.155133:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155135:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756841.155138:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155140:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756841.155142:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155144:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756841.155161:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756841.155164:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756841.155168:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756841.155170:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756841.155174:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756841.155179:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756841.155213:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756841.155215:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.155219:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756841.155222:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756841.155224:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756841.155230:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756841.155235:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756841.155239:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756841.155241:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756841.155243:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756841.155249:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.155258:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756841.155261:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:3.0:1787756841.155267:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756841.155288:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756841.155296:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756841.155305:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756841.155308:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.155311:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756841.155318:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.155323:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304568332 00000100:00000001:3.0:1787756841.155327:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304569026 00000100:00000001:3.0:1787756841.155331:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304568332 00000100:00000001:3.0:1787756841.155334:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304569026 00000100:00000001:3.0:1787756841.155337:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756841.155339:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.155342:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756841.155347:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756841/real 1787756841] req@ffff8ffb47a39180 x1874588659588608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756841.155356:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756841.155358:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756841.155368:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756841.155370:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756841.155371:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155374:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756841.155376:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756841.155380:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659588608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756841.155388:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155391:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756841.155393:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155395:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756841.155397:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155399:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756841.155418:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756841.155421:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756841.155425:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756841.155428:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756841.155432:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756841.155437:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155439:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756841.155441:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756841.155443:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756841.155445:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756841.155447:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756841.155449:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756841.155452:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756841.155453:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756841.155456:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756841.155458:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155460:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756841.155462:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756841.155463:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756841.155465:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756841.155467:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756841.155468:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756841.155470:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756841.155471:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756841.155474:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756841.155476:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155478:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756841.155479:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756841.155481:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756841.155483:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756841.155485:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756841.155486:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756841.155489:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756841.155490:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756841.155493:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756841.155495:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155497:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756841.155501:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756841.155503:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756841.155505:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756841.155507:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756841.921553:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756841.921565:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756841.921575:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756842.945541:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756842.945550:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756842.945556:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756843.969460:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756843.969467:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756843.969473:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756846.017594:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756846.017608:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756846.017618:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756846.273812:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756846.273826:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756846.273833:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756846.273839:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756846.273844:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756846.273866:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756846.273872:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756846.273901:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756846.273942:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659588736:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756846.273953:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756846.273964:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756846.273978:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756846.274003:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756846.274016:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.274044:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756846.274051:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:3.0:1787756846.274062:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756846.274122:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756846.274157:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756846.274174:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756846.274241:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.274247:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756846.274259:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.274267:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304574146 00000100:00000001:3.0:1787756846.274283:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756846.274295:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304574146 00000100:00000001:3.0:1787756846.274312:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756846.274322:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756846.274347:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.274354:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756846.274375:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756846/real 1787756846] req@ffff8ffb4790ed80 x1874588659588736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756846.274452:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756846.274463:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756846.274473:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756846.274485:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756846.274490:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.274497:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756846.274504:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756846.274514:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659588736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756846.274552:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.274565:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756846.274586:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.274600:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756846.274611:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.274631:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756846.274638:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756846.274655:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756846.274675:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756846.274690:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756846.274702:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659588736:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756846.274716:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.274735:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756846.274741:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756846.274750:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756846.274760:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756846.274769:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756846.274781:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756846.274790:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756846.274799:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756846.274805:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756846.274810:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.274826:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.274836:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756846.274838:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.274853:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756846.274861:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756846.274863:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756846.274869:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756846.274874:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756846.274884:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756846.274895:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756846.274901:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756846.274944:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588864:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756846.274955:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756846.274961:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756846.274965:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756846.274970:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756846.274977:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.274991:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756846.274994:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:3.0:1787756846.275003:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756846.275037:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756846.275045:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756846.275050:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756846.275052:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275055:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756846.275059:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275065:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304573454 00000100:00000001:3.0:1787756846.275069:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304574146 00000100:00000001:3.0:1787756846.275072:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304573454 00000100:00000001:3.0:1787756846.275076:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304574146 00000100:00000001:3.0:1787756846.275078:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756846.275081:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275084:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756846.275091:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756846/real 1787756846] req@ffff8ffb4790fb80 x1874588659588864/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756846.275103:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756846.275105:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756846.275109:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756846.275111:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756846.275114:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275117:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756846.275119:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756846.275125:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659588864/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756846.275141:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275145:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756846.275150:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275154:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756846.275156:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275159:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756846.275228:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756846.275236:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756846.275242:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756846.275246:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756846.275251:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588864:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756846.275257:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756846.275260:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756846.275262:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756846.275265:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756846.275268:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756846.275270:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756846.275275:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756846.275279:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756846.275282:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756846.275285:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756846.275287:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756846.275292:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275301:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756846.275303:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:3.0:1787756846.275308:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756846.275327:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756846.275334:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756846.275337:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756846.275340:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275342:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756846.275345:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275350:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304573452 00000100:00000001:3.0:1787756846.275353:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304574146 00000100:00000001:3.0:1787756846.275356:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304573452 00000100:00000001:3.0:1787756846.275359:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304574146 00000100:00000001:3.0:1787756846.275362:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756846.275364:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275367:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756846.275372:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756846/real 1787756846] req@ffff8ffb4790d180 x1874588659588992/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756846.275382:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756846.275384:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756846.275387:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756846.275389:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756846.275391:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275393:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756846.275395:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756846.275400:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659588992/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756846.275409:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275411:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756846.275414:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275416:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756846.275418:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275420:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756846.275434:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756846.275437:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756846.275442:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756846.275444:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756846.275449:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659588992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756846.275453:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275456:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756846.275458:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756846.275460:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756846.275462:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756846.275464:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756846.275466:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756846.275469:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756846.275471:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756846.275473:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756846.275475:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275477:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756846.275479:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756846.275480:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756846.275483:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756846.275484:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756846.275485:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756846.275487:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756846.275489:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756846.275490:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756846.275492:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275494:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275499:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756846.275501:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275503:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756846.275505:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756846.275507:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756846.275509:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756846.275511:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756846.275514:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756846.275516:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756846.275518:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756846.275524:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756846.275531:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756846.275535:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756846.275538:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756846.275542:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756846.275546:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275555:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756846.275558:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:3.0:1787756846.275565:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756846.275589:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756846.275598:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756846.275603:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756846.275607:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275610:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756846.275616:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275622:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304573452 00000100:00000001:3.0:1787756846.275627:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304574146 00000100:00000001:3.0:1787756846.275632:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304573452 00000100:00000001:3.0:1787756846.275636:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304574146 00000100:00000001:3.0:1787756846.275640:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756846.275643:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275645:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756846.275652:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756846/real 1787756846] req@ffff8ffb4790e680 x1874588659589120/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756846.275663:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756846.275665:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756846.275669:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756846.275671:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756846.275673:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275675:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756846.275677:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756846.275682:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659589120/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756846.275691:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275694:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756846.275698:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275700:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756846.275701:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275704:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756846.275718:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756846.275722:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756846.275726:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756846.275729:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756846.275733:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756846.275737:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275740:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756846.275741:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756846.275743:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756846.275745:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756846.275747:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756846.275749:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756846.275753:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756846.275754:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756846.275757:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756846.275758:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275761:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756846.275765:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756846.275767:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756846.275769:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756846.275772:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756847.041540:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756847.041546:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756847.041551:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756848.065504:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756848.065513:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756848.065519:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756850.113497:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756850.113504:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756850.113508:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756851.137535:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756851.137540:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756851.137545:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756851.393652:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756851.393663:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756851.393672:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756851.393678:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756851.393682:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756851.393697:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756851.393706:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756851.393715:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756851.393727:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659589248:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756851.393739:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756851.393749:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756851.393755:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756851.393759:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756851.393769:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.393803:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756851.393810:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:3.0:1787756851.393851:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756851.393923:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756851.393949:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756851.393959:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756851.393966:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.393971:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756851.393982:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.393993:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304579265 00000100:00000001:3.0:1787756851.394014:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756851.394026:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304579265 00000100:00000001:3.0:1787756851.394037:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756851.394047:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756851.394057:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.394064:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756851.394080:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756851/real 1787756851] req@ffff8ffb4790d880 x1874588659589248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756851.394114:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756851.394121:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756851.394136:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756851.394145:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756851.394153:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394162:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756851.394171:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756851.394242:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659589248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756851.394264:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394271:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756851.394280:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394285:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756851.394288:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394293:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756851.394299:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756851.394308:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756851.394327:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756851.394337:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756851.394349:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659589248:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756851.394364:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394369:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756851.394377:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756851.394381:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756851.394391:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756851.394398:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756851.394404:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756851.394416:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756851.394423:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756851.394429:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756851.394437:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394442:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394455:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756851.394463:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.394470:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756851.394476:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756851.394480:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756851.394485:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756851.394489:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756851.394497:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756851.394506:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756851.394510:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756851.394524:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589376:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756851.394535:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756851.394543:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756851.394549:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756851.394554:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756851.394563:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.394580:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756851.394588:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:3.0:1787756851.394599:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756851.394631:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756851.394641:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756851.394647:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756851.394651:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.394660:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756851.394667:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.394676:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304578574 00000100:00000001:3.0:1787756851.394688:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304579265 00000100:00000001:3.0:1787756851.394698:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304578574 00000100:00000001:3.0:1787756851.394702:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304579265 00000100:00000001:3.0:1787756851.394707:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756851.394720:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.394723:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756851.394735:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756851/real 1787756851] req@ffff8ffb4790f100 x1874588659589376/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756851.394795:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756851.394801:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756851.394806:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756851.394835:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756851.394838:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394842:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756851.394844:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756851.394851:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659589376/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756851.394863:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394867:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756851.394870:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394872:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756851.394875:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.394878:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756851.394900:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756851.394904:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756851.394909:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756851.394912:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756851.394917:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589376:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756851.394921:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756851.394923:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756851.394925:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756851.394927:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756851.394930:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756851.394933:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756851.394936:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756851.394941:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756851.394943:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756851.394946:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756851.394948:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756851.394953:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.394961:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756851.394964:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:3.0:1787756851.394969:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756851.394986:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756851.394992:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756851.394995:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756851.394997:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.394999:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756851.395002:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.395006:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304578571 00000100:00000001:3.0:1787756851.395010:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304579265 00000100:00000001:3.0:1787756851.395013:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304578571 00000100:00000001:3.0:1787756851.395016:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304579265 00000100:00000001:3.0:1787756851.395019:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756851.395022:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.395024:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756851.395029:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756851/real 1787756851] req@ffff8ffb4790c380 x1874588659589504/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756851.395039:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756851.395041:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756851.395043:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756851.395045:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756851.395047:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395049:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756851.395051:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756851.395056:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659589504/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756851.395065:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395067:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756851.395070:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395072:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756851.395074:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395076:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756851.395094:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756851.395097:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756851.395100:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756851.395103:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756851.395106:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756851.395110:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395113:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756851.395115:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756851.395117:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756851.395121:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756851.395124:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756851.395127:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756851.395133:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756851.395135:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756851.395139:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756851.395142:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395145:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756851.395148:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756851.395150:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756851.395153:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756851.395155:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756851.395158:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756851.395161:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756851.395163:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756851.395166:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756851.395169:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395173:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395180:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756851.395221:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.395225:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756851.395228:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756851.395231:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756851.395234:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756851.395236:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756851.395239:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756851.395242:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756851.395245:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756851.395250:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756851.395255:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756851.395258:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756851.395260:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756851.395263:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756851.395267:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.395275:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756851.395277:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:3.0:1787756851.395282:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756851.395299:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756851.395306:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756851.395309:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756851.395311:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.395314:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756851.395318:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.395322:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304578572 00000100:00000001:3.0:1787756851.395325:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304579265 00000100:00000001:3.0:1787756851.395328:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304578572 00000100:00000001:3.0:1787756851.395331:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304579265 00000100:00000001:3.0:1787756851.395334:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756851.395336:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.395339:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756851.395344:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756851/real 1787756851] req@ffff8ffb4790ce00 x1874588659589632/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756851.395353:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756851.395355:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756851.395358:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756851.395360:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756851.395362:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395364:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756851.395366:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756851.395371:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659589632/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756851.395380:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395383:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756851.395386:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395388:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756851.395390:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395392:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756851.395412:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756851.395415:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756851.395418:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756851.395420:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756851.395425:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756851.395430:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395432:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756851.395434:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756851.395436:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756851.395438:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756851.395440:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756851.395442:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756851.395445:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756851.395447:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756851.395449:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756851.395451:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395453:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756851.395457:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756851.395460:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756851.395462:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756851.395465:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756852.161349:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756852.161354:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756852.161359:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756854.209525:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756854.209531:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756854.209538:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756855.233489:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756855.233498:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756855.233505:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756856.257356:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756856.257359:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756856.257362:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756856.513442:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756856.513450:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756856.513454:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756856.513458:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756856.513461:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756856.513468:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756856.513478:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756856.513487:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756856.513497:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659589760:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756856.513505:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756856.513513:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756856.513517:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756856.513528:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756856.513538:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513567:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756856.513572:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133081856 : -123165576469760 : ffff8ffb4790e300) 00000100:00000001:3.0:1787756856.513581:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756856.513641:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756856.513664:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756856.513674:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756856.513684:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513693:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756856.513705:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513739:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304584385 00000100:00000001:3.0:1787756856.513762:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756856.513776:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304584385 00000100:00000001:3.0:1787756856.513784:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756856.513789:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756856.513794:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513796:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756856.513804:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756856/real 1787756856] req@ffff8ffb4790e300 x1874588659589760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756856.513813:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756856.513815:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756856.513820:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756856.513822:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756856.513830:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513833:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756856.513834:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756856.513838:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e300 x1874588659589760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756856.513844:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513847:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756856.513849:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513850:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756856.513851:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513853:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756856.513855:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756856.513857:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756856.513860:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756856.513862:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756856.513866:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659589760:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756856.513870:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513872:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756856.513873:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756856.513876:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756856.513879:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756856.513881:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756856.513883:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756856.513888:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756856.513889:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756856.513891:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756856.513892:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513894:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513901:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756856.513903:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513905:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756856.513907:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756856.513908:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756856.513909:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756856.513910:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756856.513912:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756856.513914:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756856.513915:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756856.513917:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589888:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756856.513920:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756856.513922:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756856.513923:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756856.513924:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756856.513926:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513931:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756856.513932:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133087232 : -123165576464384 : ffff8ffb4790f800) 00000100:00000001:3.0:1787756856.513936:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756856.513947:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756856.513951:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756856.513952:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756856.513953:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513955:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756856.513957:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513961:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304583693 00000100:00000001:3.0:1787756856.513963:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304584385 00000100:00000001:3.0:1787756856.513965:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304583693 00000100:00000001:3.0:1787756856.513967:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304584385 00000100:00000001:3.0:1787756856.513968:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756856.513969:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.513970:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756856.513973:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756856/real 1787756856] req@ffff8ffb4790f800 x1874588659589888/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756856.513978:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756856.513979:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756856.513980:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756856.513981:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756856.513982:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513983:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756856.513984:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756856.513987:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f800 x1874588659589888/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756856.513992:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513993:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756856.513995:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513996:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756856.513997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.513998:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756856.514012:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756856.514014:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756856.514016:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756856.514017:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756856.514020:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659589888:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756856.514023:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756856.514024:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756856.514025:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756856.514027:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756856.514028:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756856.514029:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756856.514031:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659590016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756856.514033:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756856.514035:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756856.514036:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756856.514037:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756856.514039:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514042:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756856.514043:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133078272 : -123165576473344 : ffff8ffb4790d500) 00000100:00000001:3.0:1787756856.514046:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756856.514052:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756856.514055:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756856.514056:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756856.514057:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514059:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756856.514060:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514063:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304583690 00000100:00000001:3.0:1787756856.514065:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304584385 00000100:00000001:3.0:1787756856.514066:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304583690 00000100:00000001:3.0:1787756856.514068:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304584385 00000100:00000001:3.0:1787756856.514069:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756856.514071:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514072:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756856.514074:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756856/real 1787756856] req@ffff8ffb4790d500 x1874588659590016/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756856.514083:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756856.514084:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756856.514086:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756856.514087:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756856.514088:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514088:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756856.514089:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756856.514091:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d500 x1874588659590016/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756856.514095:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514097:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756856.514098:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514099:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756856.514100:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514101:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756856.514114:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756856.514116:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756856.514118:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756856.514120:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756856.514122:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659590016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756856.514124:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756856.514125:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756856.514126:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756856.514128:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756856.514129:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756856.514130:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756856.514132:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659590144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756856.514134:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756856.514135:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756856.514137:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756856.514138:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756856.514140:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514142:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756856.514143:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:3.0:1787756856.514146:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756856.514152:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756856.514154:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756856.514156:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756856.514157:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514158:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756856.514159:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514162:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304583690 00000100:00000001:3.0:1787756856.514163:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304584385 00000100:00000001:3.0:1787756856.514165:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304583690 00000100:00000001:3.0:1787756856.514167:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304584385 00000100:00000001:3.0:1787756856.514169:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756856.514170:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514171:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756856.514173:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756856/real 1787756856] req@ffff8ffb4790ce00 x1874588659590144/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756856.514177:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756856.514178:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756856.514179:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756856.514180:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756856.514181:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514182:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756856.514183:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756856.514208:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659590144/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756856.514213:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514214:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756856.514216:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514217:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756856.514218:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514220:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756856.514233:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756856.514235:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756856.514237:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756856.514239:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756856.514241:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659590144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756856.514243:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514244:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756856.514245:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756856.514246:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756856.514247:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756856.514248:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756856.514249:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756856.514251:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756856.514252:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756856.514254:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756856.514255:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514256:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756856.514257:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756856.514258:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756856.514259:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756856.514260:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756856.514261:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756856.514262:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756856.514263:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756856.514264:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756856.514265:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514265:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756856.514266:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756856.514267:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756856.514268:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756856.514268:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756856.514269:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756856.514270:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756856.514271:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756856.514272:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756856.514273:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514274:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756856.514283:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756856.514285:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756856.514288:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756856.514290:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756858.305373:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756858.305377:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756858.305380:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756859.329362:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756859.329365:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756859.329368:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756860.353325:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756860.353329:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756860.353331:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756861.633436:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756861.633448:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756861.633451:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756861.633454:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756861.633459:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756861.633466:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9916 00000100:00080000:3.0:1787756861.633477:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756861.633481:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756861.633484:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756861.633490:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756861.633495:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756861.633500:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756861.633504:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756861.633507:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756861.633510:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756861.633517:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756861.633523:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756861.633528:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756861.633535:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756861.633541:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633543:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756861.633547:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756861.633562:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756861.633566:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633572:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756861.633575:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756861.633582:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633597:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756861.633609:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659590272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756861.633621:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756861.633627:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633674:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756861.633677:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756861.633682:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756861.633683:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756861.633685:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756861.633688:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756861.633691:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9916 00000100:00080000:3.0:1787756861.633696:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756861.633698:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756861.633700:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756861.633702:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756861.633703:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756861.633706:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756861.633709:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756861.633711:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756861.633712:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756861.633715:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756861.633718:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756861.633720:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756861.633726:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756861.633728:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633730:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756861.633735:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756861.633739:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633741:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756861.633746:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756861.633748:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633750:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756861.633751:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756861.633753:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633758:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756861.633766:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659590400/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756861.633775:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756861.633777:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633780:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756861.633783:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756861.633786:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756861.633788:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756861.633789:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756861.633793:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756861.633795:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9916 00000100:00080000:3.0:1787756861.633799:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756861.633801:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756861.633803:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756861.633805:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756861.633806:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756861.633809:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756861.633811:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756861.633813:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756861.633814:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756861.633818:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756861.633820:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756861.633822:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756861.633825:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756861.633828:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633830:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756861.633834:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756861.633836:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633838:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756861.633842:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756861.633844:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633846:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756861.633847:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756861.633849:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633854:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756861.633858:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659590528/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756861.633878:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756861.633880:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633883:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756861.633886:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756861.633888:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756861.633890:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756861.633892:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756861.633895:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756861.633898:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9916 00000100:00080000:3.0:1787756861.633901:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756861.633903:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756861.633905:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756861.633907:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756861.633908:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756861.633910:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756861.633912:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756861.633913:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756861.633915:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756861.633918:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756861.633920:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756861.633922:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756861.633924:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756861.633926:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633928:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756861.633931:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756861.633933:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633935:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756861.633938:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756861.633939:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633941:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756861.633942:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756861.633944:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.633948:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756861.633954:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659590656/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756861.634085:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756861.634087:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756861.634091:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756862.401319:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756862.401323:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756862.401326:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756863.425358:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756863.425366:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756863.425372:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756864.449368:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756864.449372:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756864.449376:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787756864.705470:0:2358:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1787756864.705524:0:2358:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787756864.705548:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787756864.705568:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756864.705608:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756864.705613:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756864.705616:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787756864.705630:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787756864.705638:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756864.705643:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756864.705645:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756864.705647:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787756864.705653:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787756864.705660:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756864.705664:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756864.705669:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756864.705675:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787756864.705684:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787756864.705692:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756864.705698:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756864.705704:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756864.705710:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:2.0:1787756866.497366:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756866.497373:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756866.497379:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756866.753328:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756866.753339:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756866.753342:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756866.753345:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756866.753350:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756866.753357:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9921 00000100:00080000:2.0:1787756866.753371:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756866.753375:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756866.753378:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756866.753384:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756866.753388:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756866.753391:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756866.753394:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756866.753396:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756866.753398:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756866.753404:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756866.753407:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756866.753410:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756866.753415:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756866.753419:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753422:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756866.753425:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756866.753433:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756866.753435:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753438:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756866.753441:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756866.753446:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753458:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756866.753468:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659590784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756866.753476:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756866.753481:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753499:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756866.753503:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756866.753507:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756866.753509:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756866.753511:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756866.753515:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756866.753517:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9921 00000100:00080000:2.0:1787756866.753522:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756866.753525:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756866.753528:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756866.753532:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756866.753534:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756866.753537:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756866.753540:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756866.753558:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756866.753560:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756866.753567:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756866.753573:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756866.753582:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756866.753591:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756866.753604:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753610:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756866.753624:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756866.753629:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753633:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756866.753638:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756866.753643:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753648:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756866.753652:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756866.753659:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753666:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756866.753681:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659590912/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756866.753694:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756866.753699:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753704:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756866.753708:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756866.753711:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756866.753714:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756866.753718:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756866.753726:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756866.753730:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9921 00000100:00080000:2.0:1787756866.753736:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756866.753738:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756866.753742:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756866.753752:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756866.753755:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756866.753760:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756866.753766:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756866.753771:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756866.753772:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756866.753776:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756866.753779:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756866.753782:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756866.753786:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756866.753788:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753790:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756866.753794:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756866.753798:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753800:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756866.753808:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756866.753810:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753815:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756866.753820:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756866.753826:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753836:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756866.753845:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659591040/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756866.753861:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756866.753868:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753877:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756866.753887:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756866.753895:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756866.753900:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756866.753902:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756866.753906:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756866.753912:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9921 00000100:00080000:2.0:1787756866.753921:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756866.753926:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756866.753930:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756866.753937:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756866.753940:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756866.753945:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756866.753948:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756866.753950:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756866.753952:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756866.753956:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756866.753959:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756866.753960:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756866.753965:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756866.753976:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753978:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756866.753981:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756866.753983:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.753985:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756866.753995:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756866.754001:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.754005:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756866.754009:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756866.754012:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.754025:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756866.754041:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659591168/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756866.754055:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756866.754061:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756866.754067:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756867.521350:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756867.521354:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756867.521356:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756868.545349:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756868.545353:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756868.545356:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756870.593351:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756870.593355:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756870.593358:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756871.617436:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756871.617441:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756871.617444:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756871.873275:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756871.873287:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756871.873290:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756871.873293:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756871.873298:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756871.873305:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9926 00000100:00080000:2.0:1787756871.873317:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756871.873321:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756871.873322:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756871.873326:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756871.873329:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756871.873332:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756871.873334:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756871.873336:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756871.873337:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756871.873341:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756871.873343:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756871.873345:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756871.873348:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756871.873351:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873352:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756871.873355:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756871.873367:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756871.873368:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873372:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756871.873374:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756871.873379:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873392:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756871.873401:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659591296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756871.873410:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756871.873422:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873437:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756871.873440:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756871.873442:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756871.873444:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756871.873445:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756871.873447:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756871.873449:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9926 00000100:00080000:2.0:1787756871.873465:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756871.873467:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756871.873468:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756871.873470:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756871.873471:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756871.873473:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756871.873475:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756871.873476:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756871.873477:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756871.873479:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756871.873481:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756871.873482:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756871.873487:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756871.873489:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873490:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756871.873495:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756871.873498:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873501:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756871.873505:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756871.873508:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873511:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756871.873515:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756871.873518:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873526:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756871.873535:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659591424/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756871.873549:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756871.873552:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873555:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756871.873559:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756871.873563:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756871.873566:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756871.873568:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756871.873580:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756871.873584:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9926 00000100:00080000:2.0:1787756871.873588:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756871.873589:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756871.873591:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756871.873593:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756871.873594:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756871.873595:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756871.873597:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756871.873599:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756871.873602:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756871.873606:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756871.873610:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756871.873612:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756871.873616:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756871.873618:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873619:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756871.873621:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756871.873622:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873623:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756871.873627:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756871.873628:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873630:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756871.873631:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756871.873633:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873636:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756871.873639:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659591552/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756871.873645:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756871.873647:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873648:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756871.873651:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756871.873653:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756871.873654:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756871.873655:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756871.873660:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756871.873666:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9926 00000100:00080000:2.0:1787756871.873670:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756871.873672:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756871.873673:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756871.873676:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756871.873678:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756871.873680:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756871.873682:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756871.873685:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756871.873689:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756871.873693:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756871.873696:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756871.873697:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756871.873701:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756871.873703:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873707:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756871.873708:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756871.873710:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873711:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756871.873715:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756871.873718:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873721:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756871.873722:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756871.873724:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873732:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756871.873737:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659591680/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756871.873746:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756871.873749:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756871.873751:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756872.641344:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756872.641349:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756872.641353:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756874.689369:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756874.689374:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756874.689378:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756875.713364:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756875.713367:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756875.713370:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756876.737360:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756876.737364:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756876.737382:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756876.993505:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756876.993512:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756876.993517:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756876.993521:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756876.993526:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756876.993533:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756876.993536:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756876.993539:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756876.993547:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659591808:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756876.993555:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756876.993562:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756876.993566:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756876.993572:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756876.993579:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993599:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756876.993602:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:3.0:1787756876.993610:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756876.993660:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756876.993676:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756876.993683:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756876.993687:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993691:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756876.993698:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993705:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304604865 00000100:00000001:3.0:1787756876.993720:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756876.993727:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304604865 00000100:00000001:3.0:1787756876.993735:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756876.993741:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756876.993745:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993750:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756876.993759:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756876/real 1787756876] req@ffff8ffb4790c380 x1874588659591808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756876.993773:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756876.993776:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756876.993782:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756876.993786:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756876.993790:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993793:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756876.993795:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756876.993800:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659591808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756876.993805:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993808:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756876.993810:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993811:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756876.993812:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993814:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756876.993816:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756876.993819:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756876.993822:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756876.993824:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756876.993826:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659591808:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756876.993830:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993832:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756876.993833:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756876.993835:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756876.993838:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756876.993840:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756876.993842:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756876.993846:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756876.993848:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756876.993851:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756876.993852:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993854:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993861:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756876.993863:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993865:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756876.993867:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756876.993868:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756876.993869:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756876.993870:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756876.993872:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756876.993875:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756876.993877:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756876.993880:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659591936:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756876.993882:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756876.993884:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756876.993885:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756876.993887:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756876.993890:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993895:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756876.993896:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:3.0:1787756876.993900:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756876.993913:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756876.993917:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756876.993918:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756876.993920:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993921:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756876.993923:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993926:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304604173 00000100:00000001:3.0:1787756876.993929:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304604865 00000100:00000001:3.0:1787756876.993931:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304604173 00000100:00000001:3.0:1787756876.993932:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304604865 00000100:00000001:3.0:1787756876.993934:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756876.993935:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.993937:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756876.993939:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756876/real 1787756876] req@ffff8ffb4790f100 x1874588659591936/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756876.993945:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756876.993945:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756876.993947:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756876.993948:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756876.993949:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993951:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756876.993952:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756876.993955:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659591936/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756876.993961:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993963:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756876.993966:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993967:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756876.993969:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.993971:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756876.993996:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756876.994000:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756876.994005:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756876.994008:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756876.994013:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659591936:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756876.994019:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756876.994021:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756876.994023:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756876.994025:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756876.994028:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756876.994031:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756876.994035:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756876.994039:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756876.994042:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756876.994043:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756876.994046:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756876.994050:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994056:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756876.994058:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:3.0:1787756876.994136:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756876.994153:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756876.994284:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756876.994287:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756876.994289:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994291:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756876.994294:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994329:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304604171 00000100:00000001:3.0:1787756876.994333:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304604865 00000100:00000001:3.0:1787756876.994336:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304604171 00000100:00000001:3.0:1787756876.994339:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304604865 00000100:00000001:3.0:1787756876.994342:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756876.994345:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994348:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756876.994355:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756876/real 1787756876] req@ffff8ffb4790d880 x1874588659592064/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756876.994377:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756876.994379:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756876.994383:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756876.994386:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756876.994388:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994390:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756876.994391:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756876.994394:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659592064/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756876.994399:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994401:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756876.994403:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994404:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756876.994405:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994406:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756876.994419:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756876.994420:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756876.994423:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756876.994424:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756876.994427:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756876.994430:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756876.994431:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756876.994431:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756876.994433:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756876.994434:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756876.994436:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756876.994438:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756876.994440:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756876.994442:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756876.994443:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756876.994444:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756876.994446:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994450:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756876.994451:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:3.0:1787756876.994454:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756876.994464:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756876.994467:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756876.994469:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756876.994470:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994471:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756876.994473:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994475:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304604171 00000100:00000001:3.0:1787756876.994477:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304604865 00000100:00000001:3.0:1787756876.994479:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304604171 00000100:00000001:3.0:1787756876.994480:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304604865 00000100:00000001:3.0:1787756876.994482:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756876.994483:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994484:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756876.994487:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756876/real 1787756876] req@ffff8ffb4790e680 x1874588659592192/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756876.994493:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756876.994494:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756876.994495:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756876.994497:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756876.994498:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994499:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756876.994500:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756876.994502:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659592192/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756876.994506:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994507:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756876.994509:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994510:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756876.994510:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994511:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756876.994524:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756876.994525:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756876.994527:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756876.994529:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756876.994531:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756876.994534:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994535:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756876.994536:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756876.994537:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756876.994538:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756876.994539:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756876.994540:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756876.994542:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756876.994543:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756876.994545:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756876.994546:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994547:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756876.994548:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756876.994549:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756876.994550:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756876.994551:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756876.994552:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756876.994553:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756876.994554:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756876.994554:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756876.994555:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994556:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756876.994557:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756876.994557:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756876.994558:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756876.994559:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756876.994560:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756876.994560:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756876.994561:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756876.994562:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756876.994563:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994564:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756876.994566:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756876.994567:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756876.994569:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756876.994570:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756878.785345:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756878.785350:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756878.785355:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756879.809407:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756879.809413:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756879.809417:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756880.833388:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756880.833394:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756880.833398:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756882.113309:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756882.113320:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756882.113322:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756882.113326:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756882.113331:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756882.113338:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9936 00000100:00080000:2.0:1787756882.113347:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756882.113351:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756882.113353:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756882.113357:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756882.113359:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756882.113362:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756882.113365:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756882.113367:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756882.113368:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756882.113371:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756882.113374:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113377:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756882.113380:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756882.113383:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113385:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756882.113387:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113399:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756882.113401:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113404:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756882.113407:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756882.113412:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113422:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756882.113431:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659592320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756882.113439:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756882.113443:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113456:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756882.113459:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756882.113461:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756882.113462:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756882.113463:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756882.113466:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756882.113468:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9936 00000100:00080000:2.0:1787756882.113472:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756882.113473:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756882.113475:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756882.113476:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756882.113478:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:3.0:1787756882.113478:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:2.0:1787756882.113482:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756882.113484:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787756882.113485:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756882.113487:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756882.113488:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756882.113492:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787756882.113492:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:2.0:1787756882.113496:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756882.113496:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113500:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113502:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756882.113502:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756882.113506:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756882.113507:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756882.113508:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113513:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113517:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:3.0:1787756882.113518:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659592320:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1787756882.113522:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787756882.113523:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:2.0:1787756882.113526:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756882.113529:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787756882.113530:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756882.113533:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113535:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756882.113536:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756882.113537:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113540:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756882.113541:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756882.113542:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113543:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113550:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756882.113555:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659592448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756882.113558:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756882.113561:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:2.0:1787756882.113562:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756882.113565:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756882.113566:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113571:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756882.113573:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756882.113575:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756882.113576:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756882.113577:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756882.113579:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756882.113581:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9936 00000100:00080000:2.0:1787756882.113590:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756882.113592:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756882.113593:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756882.113595:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756882.113596:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756882.113599:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000800:00000100:3.0:1787756882.113600:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787756882.113602:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756882.113605:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756882.113606:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756882.113610:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756882.113612:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756882.113613:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113616:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787756882.113618:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787756882.113619:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756882.113621:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113625:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113625:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756882.113629:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756882.113632:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787756882.113632:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787756882.113636:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113638:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304609985 00000100:00000001:2.0:1787756882.113639:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113646:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756882.113647:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113648:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756882.113649:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787756882.113650:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787756882.113651:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113658:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113658:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304609985 00000100:00100000:2.0:1787756882.113666:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659592576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756882.113666:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756882.113672:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756882.113677:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113678:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756882.113680:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787756882.113682:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113685:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756882.113688:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787756882.113688:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756882/real 1787756882] req@ffff8ffb47a39880 x1874588659592320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756882.113697:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756882.113699:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756882.113702:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756882.113707:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000400:00000001:3.0:1787756882.113709:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:2.0:1787756882.113711:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9936 00000400:00000001:3.0:1787756882.113713:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:2.0:1787756882.113715:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756882.113717:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756882.113718:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756882.113719:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756882.113720:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756882.113721:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756882.113723:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756882.113724:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756882.113725:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756882.113728:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756882.113728:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756882.113733:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113735:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787756882.113735:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:2.0:1787756882.113739:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756882.113741:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787756882.113742:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113748:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756882.113750:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787756882.113750:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00000001:2.0:1787756882.113755:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113755:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756882.113760:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756882.113763:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756882.113765:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113767:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756882.113768:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:3.0:1787756882.113769:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659592320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787756882.113770:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756882.113775:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113782:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756882.113783:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659592704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756882.113786:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756882.113792:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.113795:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756882.113798:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756882.113798:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756882.113802:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113803:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756882.113806:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756882.113807:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756882.113812:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756882.113818:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756882.113820:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756882.113823:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659592320:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756882.113827:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.113830:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756882.113831:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756882.113833:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756882.113836:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756882.113838:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756882.113839:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756882.113842:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756882.113844:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756882.113846:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756882.113847:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.113849:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.113854:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756882.113856:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113857:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756882.113858:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756882.113859:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756882.113861:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756882.113862:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756882.113863:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756882.113864:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756882.113866:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756882.113868:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592448:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756882.113870:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756882.113872:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756882.113873:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756882.113874:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756882.113877:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113882:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756882.113883:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:3.0:1787756882.113886:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756882.113896:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756882.113899:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756882.113901:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756882.113902:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113903:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756882.113905:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113908:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304609293 00000100:00000001:3.0:1787756882.113910:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304609985 00000100:00000001:3.0:1787756882.113911:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304609293 00000100:00000001:3.0:1787756882.113913:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304609985 00000100:00000001:3.0:1787756882.113915:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756882.113916:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113917:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756882.113920:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756882/real 1787756882] req@ffff8ffb47a39500 x1874588659592448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756882.113925:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756882.113926:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756882.113927:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756882.113928:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756882.113929:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.113930:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756882.113932:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756882.113934:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659592448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756882.113938:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.113940:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756882.113942:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.113943:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756882.113944:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.113945:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756882.113956:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756882.113957:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756882.113959:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756882.113961:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756882.113963:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592448:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756882.113965:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756882.113966:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756882.113967:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756882.113968:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756882.113969:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756882.113971:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756882.113972:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756882.113975:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756882.113976:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756882.113977:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756882.113978:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756882.113980:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113983:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756882.113984:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:3.0:1787756882.113986:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756882.113993:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756882.113995:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756882.113996:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756882.113998:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.113999:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756882.114000:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.114002:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304609290 00000100:00000001:3.0:1787756882.114004:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304609985 00000100:00000001:3.0:1787756882.114006:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304609290 00000100:00000001:3.0:1787756882.114008:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304609985 00000100:00000001:3.0:1787756882.114009:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756882.114010:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.114012:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756882.114014:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756882/real 1787756882] req@ffff8ffb47a3a680 x1874588659592576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756882.114018:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756882.114019:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756882.114020:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756882.114021:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756882.114022:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114023:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756882.114024:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756882.114026:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659592576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756882.114031:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114032:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756882.114033:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114035:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756882.114035:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114036:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756882.114045:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756882.114046:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756882.114048:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756882.114049:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756882.114052:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756882.114054:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756882.114055:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756882.114056:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756882.114057:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756882.114059:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756882.114060:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756882.114062:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756882.114063:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756882.114064:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756882.114065:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756882.114066:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756882.114068:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.114070:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756882.114071:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:3.0:1787756882.114073:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756882.114087:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756882.114090:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756882.114091:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756882.114092:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.114093:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756882.114095:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.114098:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304609290 00000100:00000001:3.0:1787756882.114100:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304609985 00000100:00000001:3.0:1787756882.114101:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304609290 00000100:00000001:3.0:1787756882.114103:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304609985 00000100:00000001:3.0:1787756882.114104:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756882.114105:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.114107:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756882.114109:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756882/real 1787756882] req@ffff8ffb47a38a80 x1874588659592704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756882.114113:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756882.114114:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756882.114115:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756882.114116:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756882.114117:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114119:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756882.114120:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756882.114122:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659592704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756882.114127:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114128:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756882.114129:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114130:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756882.114131:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114132:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756882.114142:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756882.114143:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756882.114145:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756882.114146:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756882.114148:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756882.114150:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114151:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756882.114152:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756882.114153:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756882.114154:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756882.114155:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756882.114156:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756882.114157:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756882.114158:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756882.114159:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756882.114160:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114161:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756882.114162:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756882.114163:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756882.114164:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756882.114165:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756882.114166:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756882.114166:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756882.114167:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756882.114169:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756882.114170:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114171:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756882.114172:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756882.114173:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756882.114174:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756882.114175:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756882.114175:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756882.114176:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756882.114177:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756882.114179:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756882.114179:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114181:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756882.114200:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756882.114201:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756882.114202:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756882.114203:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756882.881413:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756882.881421:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756882.881426:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756883.905356:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756883.905360:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756883.905363:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756884.929359:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756884.929367:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756884.929372:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756886.977399:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756886.977404:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756886.977407:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756887.233488:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756887.233499:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756887.233508:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756887.233512:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756887.233515:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756887.233526:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756887.233534:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756887.233546:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756887.233564:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659592832:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756887.233583:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756887.233603:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756887.233614:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756887.233623:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756887.233636:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.233663:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756887.233667:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:3.0:1787756887.233681:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756887.233750:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756887.233776:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756887.233785:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756887.233788:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.233792:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756887.233800:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.233805:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304615105 00000100:00000001:3.0:1787756887.233819:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756887.233826:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304615105 00000100:00000001:3.0:1787756887.233830:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756887.233833:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756887.233835:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.233838:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756887.233843:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756887/real 1787756887] req@ffff8ffb4a536300 x1874588659592832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756887.233853:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756887.233855:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756887.233860:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756887.233862:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756887.233863:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.233865:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756887.233867:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756887.233871:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659592832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756887.233876:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.233879:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756887.233882:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.233883:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756887.233884:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.233886:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756887.233888:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756887.233891:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756887.233894:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756887.233896:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756887.233899:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659592832:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756887.233903:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.233904:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756887.233906:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756887.233908:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756887.233911:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756887.233914:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756887.233916:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756887.233920:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756887.233922:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756887.233924:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756887.233925:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.233927:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.233934:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756887.233936:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.233938:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756887.233939:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756887.233940:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756887.233941:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756887.233943:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756887.233944:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756887.233946:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756887.233948:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756887.233950:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592960:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756887.233952:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756887.233954:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756887.233955:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756887.233956:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756887.233958:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.233963:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756887.233964:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:3.0:1787756887.233968:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756887.233978:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756887.233981:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756887.233983:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756887.233984:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.233985:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756887.233987:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.233991:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304614413 00000100:00000001:3.0:1787756887.233993:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304615105 00000100:00000001:3.0:1787756887.233995:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304614413 00000100:00000001:3.0:1787756887.233997:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304615105 00000100:00000001:3.0:1787756887.233999:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756887.234000:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234001:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756887.234004:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756887/real 1787756887] req@ffff8ffb4a537b80 x1874588659592960/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756887.234008:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756887.234009:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756887.234011:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756887.234012:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756887.234013:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234014:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756887.234015:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756887.234018:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659592960/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756887.234022:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234023:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756887.234025:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234026:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756887.234027:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234028:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756887.234043:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756887.234045:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756887.234048:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756887.234049:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756887.234051:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659592960:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756887.234054:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756887.234055:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756887.234056:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756887.234057:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756887.234058:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756887.234059:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756887.234061:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756887.234064:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756887.234065:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756887.234066:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756887.234068:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756887.234070:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234073:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756887.234074:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787756887.234077:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756887.234083:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756887.234085:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756887.234087:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756887.234088:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234089:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756887.234091:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234093:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304614410 00000100:00000001:3.0:1787756887.234095:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304615105 00000100:00000001:3.0:1787756887.234096:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304614410 00000100:00000001:3.0:1787756887.234098:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304615105 00000100:00000001:3.0:1787756887.234100:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756887.234101:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234103:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756887.234105:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756887/real 1787756887] req@ffff8ffb4a537800 x1874588659593088/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756887.234110:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756887.234111:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756887.234112:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756887.234113:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756887.234114:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234115:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756887.234117:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756887.234119:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659593088/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756887.234124:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234125:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756887.234126:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234127:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756887.234128:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234129:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756887.234141:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756887.234143:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756887.234145:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756887.234146:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756887.234149:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756887.234152:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756887.234154:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756887.234156:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756887.234158:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756887.234160:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756887.234162:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756887.234165:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756887.234169:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756887.234171:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756887.234172:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756887.234174:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756887.234177:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234246:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756887.234247:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:3.0:1787756887.234251:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756887.234260:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756887.234264:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756887.234266:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756887.234267:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234268:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756887.234270:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234273:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304614411 00000100:00000001:3.0:1787756887.234275:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304615105 00000100:00000001:3.0:1787756887.234277:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304614411 00000100:00000001:3.0:1787756887.234278:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304615105 00000100:00000001:3.0:1787756887.234280:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756887.234281:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234282:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756887.234285:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756887/real 1787756887] req@ffff8ffb4a535c00 x1874588659593216/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756887.234290:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756887.234291:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756887.234292:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756887.234293:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756887.234294:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234296:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756887.234297:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756887.234299:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659593216/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756887.234308:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234310:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756887.234312:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234313:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756887.234313:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234315:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756887.234328:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756887.234329:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756887.234331:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756887.234332:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756887.234334:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756887.234337:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234338:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756887.234339:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756887.234341:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756887.234342:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756887.234343:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756887.234344:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756887.234346:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756887.234347:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756887.234349:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756887.234350:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234351:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756887.234352:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756887.234353:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756887.234354:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756887.234355:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756887.234355:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756887.234357:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756887.234358:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756887.234359:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756887.234360:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234361:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756887.234361:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756887.234362:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756887.234363:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756887.234364:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756887.234365:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756887.234367:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756887.234368:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756887.234369:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756887.234370:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234371:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756887.234374:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756887.234374:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756887.234376:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756887.234377:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756888.001342:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756888.001346:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756888.001350:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756889.025357:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756889.025361:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756889.025364:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756891.073367:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756891.073370:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756891.073374:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756892.097415:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756892.097418:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756892.097421:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787756892.353474:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756892.353483:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756892.353489:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756892.353495:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756892.353500:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756892.353509:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756892.353515:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756892.353523:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756892.353533:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659593344:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756892.353541:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756892.353549:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756892.353554:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756892.353561:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756892.353580:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353602:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756892.353607:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:3.0:1787756892.353616:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756892.353664:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756892.353685:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756892.353693:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756892.353700:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353707:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756892.353718:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353729:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304620225 00000100:00000001:3.0:1787756892.353746:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756892.353758:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304620225 00000100:00000001:3.0:1787756892.353767:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756892.353771:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756892.353774:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353776:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756892.353783:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756892/real 1787756892] req@ffff8ffb4790d180 x1874588659593344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756892.353793:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756892.353795:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756892.353800:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756892.353802:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756892.353804:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.353806:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756892.353808:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756892.353811:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659593344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756892.353817:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.353820:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756892.353823:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.353824:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756892.353824:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.353827:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756892.353828:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756892.353831:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756892.353834:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756892.353836:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756892.353839:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659593344:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756892.353843:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.353845:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756892.353846:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756892.353849:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756892.353852:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756892.353854:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756892.353856:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756892.353860:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756892.353862:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756892.353864:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756892.353865:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.353868:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.353874:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756892.353876:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353878:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756892.353879:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756892.353880:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756892.353881:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756892.353882:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756892.353884:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756892.353886:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756892.353888:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756892.353891:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593472:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756892.353895:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756892.353897:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756892.353899:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756892.353901:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756892.353904:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353910:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756892.353912:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:3.0:1787756892.353917:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756892.353931:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756892.353936:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756892.353940:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756892.353941:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353943:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756892.353946:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353951:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304619533 00000100:00000001:3.0:1787756892.353955:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304620225 00000100:00000001:3.0:1787756892.353958:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304619533 00000100:00000001:3.0:1787756892.353960:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304620225 00000100:00000001:3.0:1787756892.353963:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756892.353965:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.353967:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756892.353973:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756892/real 1787756892] req@ffff8ffb4790fb80 x1874588659593472/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756892.353982:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756892.353984:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756892.353987:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756892.353989:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756892.353990:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.353992:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756892.353994:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756892.353998:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659593472/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756892.354006:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354009:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756892.354011:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354013:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756892.354015:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354016:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756892.354033:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756892.354035:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756892.354038:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756892.354040:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756892.354043:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593472:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756892.354046:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756892.354047:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756892.354048:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756892.354050:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756892.354051:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756892.354053:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756892.354056:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756892.354060:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756892.354062:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756892.354063:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756892.354066:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756892.354070:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354077:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756892.354078:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:3.0:1787756892.354082:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756892.354130:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756892.354135:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756892.354137:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756892.354138:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354140:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756892.354142:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354145:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304619530 00000100:00000001:3.0:1787756892.354147:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304620225 00000100:00000001:3.0:1787756892.354149:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304619530 00000100:00000001:3.0:1787756892.354150:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304620225 00000100:00000001:3.0:1787756892.354152:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756892.354153:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354155:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756892.354158:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756892/real 1787756892] req@ffff8ffb4790ed80 x1874588659593600/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756892.354169:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756892.354170:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756892.354172:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756892.354173:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756892.354174:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354175:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756892.354176:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756892.354179:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659593600/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756892.354206:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354208:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756892.354209:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354210:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756892.354211:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354213:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756892.354246:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756892.354250:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756892.354255:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756892.354258:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756892.354262:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756892.354267:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756892.354269:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756892.354271:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756892.354274:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756892.354277:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756892.354279:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756892.354283:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756892.354290:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756892.354293:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756892.354296:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756892.354299:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756892.354303:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354311:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756892.354312:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:3.0:1787756892.354318:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756892.354336:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756892.354359:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756892.354362:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756892.354364:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354367:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756892.354371:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354376:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304619531 00000100:00000001:3.0:1787756892.354380:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304620225 00000100:00000001:3.0:1787756892.354383:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304619531 00000100:00000001:3.0:1787756892.354385:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304620225 00000100:00000001:3.0:1787756892.354387:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756892.354389:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354391:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756892.354395:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756892/real 1787756892] req@ffff8ffb4790ca80 x1874588659593728/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756892.354403:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756892.354419:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756892.354422:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756892.354423:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756892.354424:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354425:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756892.354426:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756892.354429:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659593728/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756892.354434:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354435:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756892.354437:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354438:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756892.354439:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354440:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756892.354454:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756892.354456:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756892.354458:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756892.354460:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756892.354462:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756892.354464:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354466:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756892.354467:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756892.354468:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756892.354470:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756892.354471:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756892.354472:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756892.354474:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756892.354475:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756892.354477:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756892.354478:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354479:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756892.354480:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756892.354480:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756892.354482:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756892.354483:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756892.354483:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756892.354485:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756892.354486:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756892.354488:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756892.354489:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354491:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756892.354492:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756892.354493:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756892.354495:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756892.354496:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756892.354498:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756892.354500:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756892.354502:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756892.354504:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756892.354505:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354507:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756892.354511:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756892.354512:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756892.354514:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756892.354516:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756893.121345:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756893.121350:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756893.121354:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756895.169357:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756895.169364:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756895.169368:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756896.193482:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756896.193490:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756896.193498:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756897.217434:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756897.217441:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756897.217449:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756897.473510:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756897.473532:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756897.473538:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756897.473545:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756897.473554:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756897.473569:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9952 00000100:00080000:2.0:1787756897.473591:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756897.473599:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756897.473604:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756897.473613:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756897.473619:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756897.473626:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756897.473633:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756897.473637:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756897.473640:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756897.473650:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756897.473657:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756897.473663:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756897.473671:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756897.473677:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.473682:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756897.473689:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756897.473708:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756897.473713:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.473721:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756897.473727:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756897.473736:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.473761:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756897.473784:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659593856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756897.473805:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756897.473813:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.473847:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756897.473854:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756897.473862:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756897.473866:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756897.473868:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756897.473874:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756897.473878:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9952 00000100:00080000:2.0:1787756897.473885:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756897.473889:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756897.473893:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756897.473898:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756897.473902:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756897.473906:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756897.473911:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756897.473916:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756897.473919:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756897.473929:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756897.473935:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756897.473937:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756897.473945:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787756897.473951:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787756897.473961:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756897.473962:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756897.473972:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787756897.473974:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.473980:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.473983:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:3.0:1787756897.473996:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00100000:2.0:1787756897.473998:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787756897.474032:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:2.0:1787756897.474037:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.474045:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.474047:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756897.474064:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:3.0:1787756897.474066:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659593856:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787756897.474070:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.474080:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:3.0:1787756897.474085:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756897.474088:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756897.474097:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.474098:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756897.474107:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.474112:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.474114:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756897.474130:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756897.474135:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659593984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756897.474156:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756897.474159:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474164:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756897.474165:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474175:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00080000:2.0:1787756897.474176:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756897.474240:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756897.474248:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756897.474256:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756897.474256:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787756897.474269:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756897.474277:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9952 00000100:00080000:2.0:1787756897.474342:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756897.474348:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756897.474352:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:3.0:1787756897.474361:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787756897.474362:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756897.474372:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756897.474377:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756897.474383:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756897.474389:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756897.474392:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756897.474393:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787756897.474408:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:2.0:1787756897.474410:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756897.474417:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.474422:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474426:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787756897.474433:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787756897.474443:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756897.474444:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756897.474452:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.474456:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756897.474461:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756897.474474:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.474479:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474485:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304625346 02000000:00000001:2.0:1787756897.474490:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756897.474497:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.474504:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756897.474507:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756897.474511:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474517:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756897.474524:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474533:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304625346 00000100:00100000:2.0:1787756897.474544:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659594112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756897.474552:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756897.474565:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756897.474574:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756897.474577:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756897.474585:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474586:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787756897.474598:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756897.474603:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756897/real 1787756897] req@ffff8ffb47a38380 x1874588659593856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756897.474608:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756897.474628:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756897.474635:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756897.474642:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000400:00000001:3.0:1787756897.474643:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756897.474652:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756897.474655:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756897.474666:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9952 00000100:00000001:3.0:1787756897.474668:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:2.0:1787756897.474678:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756897.474680:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756897.474687:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756897.474690:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756897.474699:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756897.474700:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787756897.474709:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787756897.474711:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756897.474718:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756897.474724:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:3.0:1787756897.474731:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659593856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787756897.474732:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756897.474747:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756897.474757:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.474762:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756897.474766:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787756897.474769:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.474779:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:2.0:1787756897.474786:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756897.474787:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.474800:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756897.474802:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474809:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:2.0:1787756897.474811:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756897.474821:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787756897.474823:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:2.0:1787756897.474835:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756897.474844:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756897.474847:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756897.474855:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787756897.474858:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000008:00000001:2.0:1787756897.474867:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756897.474872:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659593856:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756897.474877:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474884:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787756897.474890:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787756897.474891:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787756897.474897:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474899:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:2.0:1787756897.474904:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756897.474907:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756897.474909:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756897.474915:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.474916:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756897.474922:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:2.0:1787756897.474925:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756897.474926:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756897.474936:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756897.474940:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00100000:2.0:1787756897.474943:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659594240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756897.474948:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756897.474953:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.474961:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756897.474973:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756897.474976:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474980:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756897.474982:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474986:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.474993:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756897.474997:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756897.475028:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756897.475032:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756897.475035:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.475042:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756897.475047:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756897.475052:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756897.475061:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593984:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756897.475068:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756897.475072:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756897.475076:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.475079:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756897.475086:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475099:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756897.475102:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:3.0:1787756897.475112:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756897.475141:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756897.475151:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756897.475155:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756897.475157:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475160:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756897.475164:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475171:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304624654 00000100:00000001:3.0:1787756897.475176:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304625346 00000100:00000001:3.0:1787756897.475181:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304624654 00000100:00000001:3.0:1787756897.475227:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304625346 00000100:00000001:3.0:1787756897.475233:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756897.475238:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475242:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756897.475253:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756897/real 1787756897] req@ffff8ffb47a38000 x1874588659593984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756897.475272:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756897.475275:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756897.475281:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756897.475285:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756897.475288:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475292:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756897.475295:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756897.475304:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659593984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756897.475320:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475324:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756897.475329:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475333:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756897.475336:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475341:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756897.475368:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756897.475374:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756897.475381:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756897.475386:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756897.475393:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659593984:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756897.475402:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756897.475405:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756897.475408:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.475413:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756897.475417:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756897.475422:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756897.475428:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756897.475435:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756897.475439:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756897.475442:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.475447:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756897.475453:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475466:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756897.475469:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:3.0:1787756897.475478:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756897.475505:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756897.475515:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756897.475520:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756897.475523:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475527:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756897.475533:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475540:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304624652 00000100:00000001:3.0:1787756897.475545:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304625346 00000100:00000001:3.0:1787756897.475549:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304624652 00000100:00000001:3.0:1787756897.475552:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304625346 00000100:00000001:3.0:1787756897.475555:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756897.475557:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475560:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756897.475567:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756897/real 1787756897] req@ffff8ffb47a3b480 x1874588659594112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756897.475579:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756897.475581:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756897.475585:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756897.475587:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756897.475589:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475591:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756897.475593:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756897.475598:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659594112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756897.475608:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475611:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756897.475614:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475616:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756897.475617:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475619:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756897.475639:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756897.475644:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756897.475649:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756897.475653:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756897.475660:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756897.475668:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756897.475670:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756897.475673:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.475679:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756897.475682:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756897.475687:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756897.475693:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756897.475700:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756897.475703:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756897.475707:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756897.475710:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756897.475716:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475725:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756897.475727:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:3.0:1787756897.475733:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756897.475751:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756897.475757:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756897.475760:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756897.475763:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475766:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756897.475769:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475775:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304624652 00000100:00000001:3.0:1787756897.475780:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304625346 00000100:00000001:3.0:1787756897.475784:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304624652 00000100:00000001:3.0:1787756897.475789:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304625346 00000100:00000001:3.0:1787756897.475793:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756897.475796:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.475800:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756897.475808:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756897/real 1787756897] req@ffff8ffb47a39180 x1874588659594240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756897.475825:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756897.475828:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756897.475833:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756897.475836:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756897.475839:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475843:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756897.475846:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756897.475855:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659594240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756897.475867:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475870:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756897.475873:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475876:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756897.475878:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475880:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756897.475900:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756897.475903:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756897.475908:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756897.475911:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756897.475916:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756897.475921:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475924:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756897.475926:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756897.475928:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756897.475931:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756897.475933:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756897.475935:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756897.475938:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756897.475950:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756897.475953:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756897.475955:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475957:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756897.475959:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756897.475960:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756897.475963:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756897.475964:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756897.475966:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756897.475967:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756897.475969:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756897.475972:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756897.475973:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475975:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756897.475977:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756897.475978:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756897.475979:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756897.475981:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756897.475982:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756897.475984:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756897.475985:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756897.475989:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756897.475990:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475992:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756897.475997:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756897.475999:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756897.476028:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756897.476031:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756899.265560:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756899.265567:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756899.265572:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756900.289503:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756900.289521:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756900.289527:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756901.313468:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756901.313475:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756901.313480:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756902.593856:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756902.593875:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756902.593880:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756902.593886:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756902.593895:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756902.593907:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9957 00000100:00080000:2.0:1787756902.593974:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756902.593982:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756902.593986:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756902.593994:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756902.593999:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756902.594004:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756902.594010:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756902.594014:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756902.594017:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756902.594027:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756902.594033:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756902.594039:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756902.594046:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756902.594052:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594056:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756902.594079:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756902.594098:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756902.594102:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594108:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756902.594122:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756902.594129:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594150:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756902.594167:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659594368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756902.594249:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756902.594257:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594282:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756902.594288:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756902.594294:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756902.594297:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756902.594300:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756902.594306:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756902.594311:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9957 00000100:00080000:2.0:1787756902.594319:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756902.594324:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756902.594326:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756902.594329:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756902.594331:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756902.594334:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756902.594337:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756902.594339:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756902.594341:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756902.594345:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756902.594349:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756902.594351:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756902.594357:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756902.594360:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594362:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756902.594368:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756902.594372:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594374:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756902.594381:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756902.594383:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594385:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756902.594387:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756902.594390:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594397:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756902.594405:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659594496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756902.594420:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756902.594425:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594429:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756902.594433:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756902.594439:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756902.594442:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756902.594446:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756902.594451:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756902.594456:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9957 00000100:00080000:2.0:1787756902.594462:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756902.594466:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756902.594469:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756902.594473:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756902.594475:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756902.594479:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756902.594483:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756902.594487:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756902.594489:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756902.594496:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756902.594501:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756902.594505:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756902.594509:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756902.594513:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594517:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756902.594521:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756902.594525:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594528:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756902.594536:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756902.594539:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594542:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756902.594544:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756902.594546:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594553:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756902.594560:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659594624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756902.594570:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756902.594573:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594576:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756902.594691:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756902.594699:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756902.594712:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756902.594717:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756902.594722:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756902.594729:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9957 00000100:00080000:2.0:1787756902.594738:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756902.594742:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756902.594751:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756902.594768:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787756902.594769:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:2.0:1787756902.594773:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756902.594780:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756902.594786:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787756902.594790:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756902.594796:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756902.594800:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756902.594805:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756902.594808:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787756902.594814:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756902.594823:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756902.594829:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.594831:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756902.594833:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756902.594846:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756902.594847:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756902.594861:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594866:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:3.0:1787756902.594867:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:2.0:1787756902.594872:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756902.594878:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.594880:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756902.594889:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756902.594892:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756902.594892:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659594368:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756902.594899:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756902.594906:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756902.594933:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.594940:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787756902.594947:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.594954:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756902.594963:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756902.594966:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659594752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787756902.594973:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787756902.594999:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756902.595005:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.595007:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756902.595019:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595047:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756902.595051:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:3.0:1787756902.595059:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756902.595124:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756902.595142:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756902.595149:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756902.595152:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595155:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756902.595163:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595169:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304630466 00000100:00000001:3.0:1787756902.595185:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787756902.595194:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304630466 00000100:00000001:3.0:1787756902.595250:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787756902.595255:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756902.595259:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595263:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756902.595273:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756902/real 1787756902] req@ffff8ffb47a38700 x1874588659594368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756902.595300:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756902.595320:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756902.595328:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756902.595331:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756902.595333:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595337:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756902.595340:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756902.595347:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659594368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756902.595358:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595362:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756902.595367:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595368:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756902.595370:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595374:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756902.595376:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756902.595381:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756902.595386:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756902.595389:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756902.595395:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659594368:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787756902.595401:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756902.595403:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756902.595405:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.595409:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756902.595411:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756902.595414:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756902.595418:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594496:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756902.595424:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756902.595427:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756902.595430:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.595433:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756902.595437:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595445:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756902.595447:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:3.0:1787756902.595454:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756902.595472:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756902.595479:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756902.595482:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756902.595484:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595486:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756902.595489:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595514:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304629775 00000100:00000001:3.0:1787756902.595518:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304630466 00000100:00000001:3.0:1787756902.595521:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304629775 00000100:00000001:3.0:1787756902.595524:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304630466 00000100:00000001:3.0:1787756902.595527:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756902.595530:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595532:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756902.595537:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756902/real 1787756902] req@ffff8ffb47a3a300 x1874588659594496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756902.595546:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756902.595548:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756902.595551:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756902.595553:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756902.595554:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595556:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756902.595558:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756902.595563:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659594496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756902.595571:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595574:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756902.595596:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595598:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756902.595600:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595602:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756902.595625:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756902.595628:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756902.595633:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756902.595636:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756902.595640:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594496:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787756902.595645:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756902.595647:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756902.595649:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.595652:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756902.595655:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756902.595657:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756902.595661:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756902.595665:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756902.595668:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756902.595671:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.595673:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756902.595677:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595685:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756902.595687:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:3.0:1787756902.595692:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756902.595705:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756902.595710:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756902.595713:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756902.595715:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595717:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756902.595721:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595725:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304629772 00000100:00000001:3.0:1787756902.595728:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304630466 00000100:00000001:3.0:1787756902.595732:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304629772 00000100:00000001:3.0:1787756902.595735:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304630466 00000100:00000001:3.0:1787756902.595738:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756902.595740:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.595743:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756902.595747:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756902/real 1787756902] req@ffff8ffb47a39180 x1874588659594624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756902.595774:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756902.595775:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756902.595778:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756902.595779:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756902.595781:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595783:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756902.595784:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756902.595788:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659594624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756902.595797:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595799:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756902.595802:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595805:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756902.595806:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595808:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756902.595826:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756902.595828:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756902.595832:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756902.595834:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756902.595838:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756902.595861:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595864:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756902.595867:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756902.595871:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756902.595876:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756902.595880:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756902.595883:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756902.595891:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756902.595893:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756902.595898:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756902.595900:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595902:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756902.595904:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756902.595906:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756902.595908:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756902.595937:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756902.595939:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756902.595960:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756902.595963:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756902.595967:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756902.595969:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595971:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756902.595973:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756902.595975:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756902.595976:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756902.595978:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756902.595979:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756902.595981:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756902.595983:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756902.595986:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756902.595988:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.595991:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.596002:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756902.596004:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.596008:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756902.596010:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787756902.596011:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787756902.596014:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787756902.596015:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.596019:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787756902.596021:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756902.596024:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756902.596028:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756902.596036:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787756902.596039:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787756902.596041:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756902.596043:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787756902.596047:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.596054:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787756902.596055:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:3.0:1787756902.596061:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787756902.596076:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787756902.596081:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756902.596084:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756902.596086:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.596088:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787756902.596091:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.596096:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304629772 00000100:00000001:3.0:1787756902.596099:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304630466 00000100:00000001:3.0:1787756902.596102:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304629772 00000100:00000001:3.0:1787756902.596105:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304630466 00000100:00000001:3.0:1787756902.596107:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787756902.596109:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.596111:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787756902.596116:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756902/real 1787756902] req@ffff8ffb47a3b480 x1874588659594752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787756902.596125:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787756902.596126:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787756902.596128:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787756902.596130:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787756902.596132:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.596134:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756902.596136:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787756902.596140:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659594752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787756902.596156:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.596159:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787756902.596162:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.596164:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787756902.596165:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.596167:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787756902.596187:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787756902.596190:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756902.596193:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787756902.596195:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787756902.596223:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659594752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787756902.596229:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.596231:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756902.596233:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787756902.596235:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787756902.596238:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787756902.596239:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787756902.596241:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787756902.596244:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787756902.596245:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787756902.596248:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787756902.596250:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.596252:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787756902.596256:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787756902.596258:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756902.596260:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787756902.596263:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756904.385502:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756904.385512:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756904.385519:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756905.409537:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756905.409549:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756905.409558:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756907.713420:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756907.713435:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756907.713440:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756907.713446:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756907.713452:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756907.713462:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9962 00000100:00080000:3.0:1787756907.713479:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756907.713483:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756907.713486:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756907.713491:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756907.713495:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756907.713500:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756907.713503:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756907.713505:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756907.713507:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756907.713512:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756907.713515:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756907.713518:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756907.713523:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756907.713527:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713530:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756907.713533:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756907.713552:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756907.713555:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713559:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756907.713563:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756907.713570:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713588:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756907.713607:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659594880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756907.713626:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756907.713634:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713670:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756907.713677:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756907.713687:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756907.713690:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756907.713693:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756907.713698:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756907.713703:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9962 00000100:00080000:3.0:1787756907.713710:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756907.713713:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756907.713715:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756907.713719:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756907.713721:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756907.713725:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756907.713727:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756907.713730:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756907.713732:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756907.713738:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756907.713742:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756907.713747:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756907.713755:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756907.713763:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713768:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756907.713779:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756907.713788:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713791:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756907.713797:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756907.713801:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713804:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756907.713809:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756907.713817:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713861:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756907.713876:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659595008/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756907.713907:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756907.713912:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713916:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756907.713924:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756907.713928:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756907.713930:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756907.713932:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756907.713936:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756907.713940:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9962 00000100:00080000:3.0:1787756907.713946:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756907.713955:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756907.713960:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756907.713967:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756907.713969:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756907.713972:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756907.713975:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756907.713977:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756907.713978:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756907.713983:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756907.713985:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756907.713987:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756907.713991:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756907.713993:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.713995:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756907.713999:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756907.714002:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714005:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756907.714012:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756907.714013:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714016:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756907.714018:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756907.714021:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714031:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756907.714041:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659595136/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756907.714051:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756907.714056:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714061:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756907.714066:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756907.714073:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756907.714078:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756907.714080:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756907.714085:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756907.714091:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9962 00000100:00080000:3.0:1787756907.714100:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756907.714103:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756907.714104:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756907.714107:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756907.714108:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756907.714111:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756907.714118:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756907.714121:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756907.714125:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756907.714137:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756907.714148:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756907.714153:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756907.714159:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756907.714166:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714171:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756907.714176:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756907.714259:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714265:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756907.714277:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756907.714282:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714288:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756907.714294:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756907.714297:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714306:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756907.714325:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659595264/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756907.714342:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756907.714348:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756907.714355:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756908.481510:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756908.481516:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756908.481522:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756909.505475:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756909.505484:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756909.505504:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756912.577604:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756912.577611:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756912.577616:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756912.833483:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756912.833494:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756912.833511:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756912.833518:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756912.833526:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756912.833537:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9967 00000100:00080000:3.0:1787756912.833552:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756912.833558:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756912.833563:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756912.833570:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756912.833575:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756912.833581:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756912.833586:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756912.833591:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756912.833595:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756912.833603:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756912.833609:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756912.833615:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756912.833623:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756912.833630:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.833634:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756912.833639:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756912.833684:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756912.833688:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.833695:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756912.833700:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756912.833708:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.833726:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756912.833786:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659595392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756912.833806:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756912.833814:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.833840:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756912.833849:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756912.833858:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756912.833863:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756912.833866:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756912.833873:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756912.833880:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9967 00000100:00080000:3.0:1787756912.833893:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756912.833898:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756912.833902:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756912.833906:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756912.833909:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756912.833914:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756912.833921:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756912.833937:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756912.833944:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756912.833964:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756912.833976:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756912.834181:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756912.834608:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756912.834625:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.834653:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756912.834664:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756912.834672:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.834677:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756912.834690:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756912.834696:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.834702:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756912.834707:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756912.834713:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.834727:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756912.834785:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659595520/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756912.834800:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756912.834804:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.834824:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756912.834830:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756912.834836:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756912.834839:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756912.834842:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756912.834847:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756912.834852:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9967 00000100:00080000:3.0:1787756912.834861:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756912.834866:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756912.834869:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756912.834875:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756912.834878:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756912.834882:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756912.834888:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756912.834892:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756912.834895:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756912.834914:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756912.834931:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756912.834944:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756912.834955:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756912.834964:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.834969:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756912.834974:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756912.834986:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.834995:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756912.835011:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756912.835017:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.835046:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756912.835050:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756912.835064:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.835075:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756912.835103:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659595648/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756912.835123:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756912.835132:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.835148:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756912.835160:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756912.835171:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756912.835178:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756912.835184:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756912.835208:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756912.835216:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9967 00000100:00080000:3.0:1787756912.835273:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756912.835289:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756912.835298:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756912.835323:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756912.835332:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756912.835340:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756912.835355:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756912.835373:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756912.835381:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756912.835409:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756912.835429:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756912.835441:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756912.835453:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756912.835459:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.835463:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756912.835468:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756912.835472:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.835476:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756912.835483:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756912.835488:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.835505:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756912.835517:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756912.835530:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.835555:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756912.835580:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659595776/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756912.835616:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756912.835632:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756912.835645:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756913.601492:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756913.601502:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756913.601509:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756916.673546:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756916.673552:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756916.673557:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756917.697547:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756917.697553:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756917.697558:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756917.953345:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756917.953361:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756917.953365:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756917.953371:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756917.953378:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756917.953389:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9972 00000100:00080000:3.0:1787756917.953405:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756917.953410:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756917.953413:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756917.953419:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756917.953423:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756917.953428:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756917.953431:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756917.953434:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756917.953436:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756917.953443:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756917.953446:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756917.953450:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756917.953455:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756917.953459:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953462:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756917.953465:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756917.953485:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756917.953487:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953492:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756917.953497:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756917.953504:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953534:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756917.953549:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659595904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756917.953565:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756917.953571:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953597:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756917.953602:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756917.953607:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756917.953609:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756917.953612:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756917.953615:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756917.953618:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9972 00000100:00080000:3.0:1787756917.953623:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756917.953625:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756917.953627:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756917.953630:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756917.953632:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756917.953635:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756917.953660:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756917.953665:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756917.953678:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756917.953684:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756917.953687:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756917.953690:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756917.953699:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756917.953703:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953711:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756917.953719:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756917.953727:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953735:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756917.953743:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756917.953748:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953757:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756917.953759:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756917.953763:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953773:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756917.953795:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659596032/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756917.953822:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756917.953828:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953839:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756917.953847:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756917.953852:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756917.953860:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756917.953864:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756917.953868:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756917.953873:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9972 00000100:00080000:3.0:1787756917.953878:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756917.953880:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756917.953884:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756917.953891:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756917.953897:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756917.953902:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756917.953910:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756917.953913:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756917.953915:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756917.953919:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756917.953922:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756917.953924:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756917.953934:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756917.953936:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953939:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756917.953943:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756917.953946:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953948:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756917.953953:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756917.953955:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953958:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756917.953960:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756917.953963:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.953972:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756917.953978:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659596160/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756917.954000:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756917.954003:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.954008:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756917.954012:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756917.954018:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756917.954025:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756917.954028:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756917.954033:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756917.954038:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9972 00000100:00080000:3.0:1787756917.954043:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756917.954045:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756917.954047:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756917.954049:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756917.954050:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756917.954056:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756917.954060:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756917.954067:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756917.954073:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756917.954078:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756917.954083:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756917.954088:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756917.954093:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756917.954103:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.954106:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756917.954110:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756917.954117:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.954125:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756917.954137:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756917.954146:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.954159:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756917.954169:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756917.954179:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.954256:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756917.954275:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659596288/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756917.954294:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756917.954301:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756917.954309:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756920.769428:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756920.769433:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756920.769438:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756921.793498:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756921.793512:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756921.793521:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756923.073329:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756923.073347:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756923.073351:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756923.073357:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756923.073365:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756923.073376:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 9977 00000100:00080000:3.0:1787756923.073394:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756923.073399:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756923.073403:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756923.073409:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756923.073414:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756923.073418:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756923.073422:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756923.073425:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756923.073427:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756923.073434:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756923.073439:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756923.073443:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756923.073448:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756923.073453:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073456:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756923.073460:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756923.073478:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756923.073481:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073486:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756923.073490:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756923.073497:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073517:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756923.073532:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659596416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756923.073577:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756923.073584:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073608:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756923.073613:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756923.073618:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756923.073620:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756923.073622:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756923.073626:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756923.073630:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9977 00000100:00080000:3.0:1787756923.073635:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756923.073639:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756923.073642:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756923.073645:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756923.073648:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756923.073652:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756923.073655:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756923.073658:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756923.073660:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756923.073666:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756923.073670:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756923.073678:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756923.073691:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756923.073701:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073709:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756923.073715:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756923.073725:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073730:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756923.073739:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756923.073743:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073753:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756923.073758:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756923.073766:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073774:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756923.073792:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659596544/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756923.073811:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756923.073815:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073819:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756923.073823:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756923.073827:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756923.073830:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756923.073840:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756923.073851:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756923.073858:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9977 00000100:00080000:3.0:1787756923.073866:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756923.073869:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756923.073870:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756923.073873:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756923.073874:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756923.073878:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756923.073882:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756923.073884:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756923.073887:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756923.073891:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756923.073895:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756923.073896:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756923.073900:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756923.073902:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073905:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756923.073909:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756923.073912:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073915:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756923.073921:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756923.073923:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073925:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756923.073926:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756923.073928:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073936:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756923.073950:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659596672/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756923.073966:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756923.073969:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.073973:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756923.073979:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756923.073988:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756923.073991:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756923.073995:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756923.074001:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756923.074010:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 9977 00000100:00080000:3.0:1787756923.074032:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756923.074043:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756923.074049:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756923.074052:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756923.074053:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756923.074056:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756923.074060:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756923.074069:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756923.074071:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756923.074077:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756923.074085:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756923.074088:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756923.074098:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756923.074104:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.074110:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756923.074117:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756923.074124:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.074127:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756923.074136:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756923.074141:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.074149:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756923.074153:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756923.074158:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.074169:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756923.074214:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659596800/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756923.074226:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756923.074229:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756923.074234:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756924.865390:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756924.865395:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756924.865399:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756925.889545:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756925.889551:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756925.889557:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756928.961632:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756928.961638:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756928.961643:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756929.985500:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756929.985509:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756929.985516:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787756931.265631:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787756931.265700:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787756931.265745:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756931.265768:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756931.265796:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756931.265800:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756931.265803:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756931.265825:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756931.265836:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756931.265843:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756931.265845:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756931.265847:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756931.265855:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756931.265861:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756931.265867:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756931.265873:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756931.265882:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787756931.265896:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787756931.265909:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787756931.265917:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787756931.265921:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756931.265928:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:3.0:1787756933.057513:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756933.057519:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756933.057525:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756934.081482:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756934.081491:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756934.081498:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756937.153564:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756937.153571:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756937.153576:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756938.177523:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756938.177529:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756938.177534:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756938.434590:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756938.434602:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756938.434612:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756938.434622:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756938.434640:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756938.434655:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756938.434663:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756938.434677:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756938.434695:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659597952:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756938.434714:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756938.434724:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756938.434732:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756938.434739:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756938.434754:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.434790:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756938.434799:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133072896 : -123165576478720 : ffff8ffb4790c000) 00000100:00000001:2.0:1787756938.434815:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756938.434899:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756938.434922:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756938.434936:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756938.434947:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.434959:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756938.434975:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.434991:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304666306 00000100:00000001:2.0:1787756938.435013:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756938.435029:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304666306 00000100:00000001:2.0:1787756938.435045:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756938.435057:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756938.435069:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435079:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756938.435101:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756938/real 1787756938] req@ffff8ffb4790c000 x1874588659597952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756938.435149:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756938.435155:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756938.435165:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756938.435174:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756938.435181:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435298:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756938.435304:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756938.435320:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c000 x1874588659597952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756938.435333:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435338:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756938.435343:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435346:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756938.435348:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435351:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756938.435354:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756938.435358:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.435363:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756938.435366:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756938.435372:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659597952:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756938.435392:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435396:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756938.435398:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756938.435401:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756938.435409:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756938.435416:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756938.435422:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756938.435429:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756938.435434:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756938.435438:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756938.435441:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435449:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435458:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756938.435462:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435466:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756938.435478:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756938.435483:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756938.435488:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756938.435492:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756938.435499:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756938.435502:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756938.435506:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756938.435515:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598080:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756938.435526:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756938.435530:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756938.435534:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756938.435539:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756938.435546:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435561:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756938.435565:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133080064 : -123165576471552 : ffff8ffb4790dc00) 00000100:00000001:2.0:1787756938.435573:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756938.435606:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756938.435617:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756938.435623:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756938.435626:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435629:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756938.435634:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435640:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304665615 00000100:00000001:2.0:1787756938.435645:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304666306 00000100:00000001:2.0:1787756938.435648:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304665615 00000100:00000001:2.0:1787756938.435651:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304666306 00000100:00000001:2.0:1787756938.435654:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756938.435656:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435659:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756938.435666:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756938/real 1787756938] req@ffff8ffb4790dc00 x1874588659598080/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756938.435679:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756938.435681:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756938.435685:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756938.435687:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756938.435689:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435691:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756938.435693:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756938.435699:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790dc00 x1874588659598080/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756938.435717:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435720:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756938.435724:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435726:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756938.435727:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435730:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756938.435746:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756938.435749:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.435754:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756938.435756:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756938.435760:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598080:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756938.435765:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756938.435767:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756938.435769:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756938.435772:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756938.435775:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756938.435778:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756938.435781:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756938.435785:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756938.435788:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756938.435790:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756938.435793:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756938.435796:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435803:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756938.435805:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:2.0:1787756938.435810:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756938.435825:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756938.435830:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756938.435833:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756938.435834:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435837:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756938.435840:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435844:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304665612 00000100:00000001:2.0:1787756938.435847:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304666306 00000100:00000001:2.0:1787756938.435850:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304665612 00000100:00000001:2.0:1787756938.435853:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304666306 00000100:00000001:2.0:1787756938.435856:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756938.435858:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435860:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756938.435865:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756938/real 1787756938] req@ffff8ffb4790f480 x1874588659598208/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756938.435875:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756938.435876:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756938.435879:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756938.435881:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756938.435883:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435885:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756938.435886:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756938.435891:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659598208/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756938.435899:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435902:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756938.435905:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435907:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756938.435908:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435910:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756938.435924:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756938.435927:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.435931:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756938.435933:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756938.435938:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756938.435943:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435945:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756938.435947:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756938.435949:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756938.435952:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756938.435954:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756938.435955:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756938.435958:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756938.435960:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756938.435963:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756938.435964:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435966:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756938.435968:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756938.435970:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756938.435971:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756938.435973:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756938.435975:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756938.435976:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756938.435977:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756938.435980:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756938.435981:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435983:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.435988:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756938.435989:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.435992:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756938.435994:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756938.435995:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756938.435997:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756938.435999:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756938.436001:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756938.436004:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756938.436006:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756938.436010:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756938.436015:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756938.436019:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756938.436022:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756938.436026:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756938.436032:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.436041:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756938.436044:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:2.0:1787756938.436051:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756938.436074:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756938.436083:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756938.436087:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756938.436089:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.436093:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756938.436098:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.436104:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304665612 00000100:00000001:2.0:1787756938.436109:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304666306 00000100:00000001:2.0:1787756938.436112:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304665612 00000100:00000001:2.0:1787756938.436115:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304666306 00000100:00000001:2.0:1787756938.436118:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756938.436120:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.436123:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756938.436129:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756938/real 1787756938] req@ffff8ffb4790ca80 x1874588659598336/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756938.436140:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756938.436141:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756938.436145:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756938.436147:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756938.436149:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.436152:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756938.436153:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756938.436158:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659598336/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756938.436167:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.436170:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756938.436172:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.436174:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756938.436177:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.436179:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756938.436233:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756938.436240:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.436248:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756938.436253:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756938.436260:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756938.436267:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.436270:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756938.436272:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756938.436274:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756938.436301:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756938.436304:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756938.436306:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756938.436309:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756938.436311:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756938.436314:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756938.436315:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.436318:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756938.436323:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756938.436325:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756938.436327:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756938.436330:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756941.249549:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756941.249554:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756941.249558:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756942.273464:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756942.273470:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756942.273475:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756943.553736:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756943.553746:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756943.553755:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756943.553763:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756943.553771:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756943.553785:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756943.553792:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756943.553802:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756943.553818:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659598464:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756943.553843:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756943.553868:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756943.553876:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756943.553886:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756943.553900:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.553930:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756943.553936:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:2.0:1787756943.553947:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756943.554016:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756943.554042:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756943.554054:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756943.554064:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554070:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756943.554084:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554096:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304671425 00000100:00000001:2.0:1787756943.554115:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756943.554129:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304671425 00000100:00000001:2.0:1787756943.554140:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756943.554162:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756943.554172:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554179:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756943.554255:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756943/real 1787756943] req@ffff8ffb4790ed80 x1874588659598464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756943.554279:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756943.554289:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756943.554300:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756943.554311:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756943.554318:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554325:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756943.554332:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756943.554345:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659598464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756943.554369:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554380:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756943.554387:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554398:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756943.554403:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554409:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756943.554415:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756943.554425:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756943.554437:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756943.554443:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756943.554456:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659598464:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756943.554469:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554478:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756943.554483:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756943.554488:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756943.554497:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756943.554504:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756943.554511:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756943.554521:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756943.554526:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756943.554534:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756943.554540:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554547:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554558:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756943.554562:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554566:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756943.554569:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756943.554572:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756943.554574:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756943.554577:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756943.554579:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756943.554583:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756943.554586:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756943.554592:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598592:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756943.554597:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756943.554601:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756943.554605:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756943.554609:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756943.554615:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554626:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756943.554629:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:2.0:1787756943.554638:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756943.554665:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756943.554675:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756943.554680:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756943.554683:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554687:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756943.554692:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554701:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304670734 00000100:00000001:2.0:1787756943.554707:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304671425 00000100:00000001:2.0:1787756943.554712:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304670734 00000100:00000001:2.0:1787756943.554717:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304671425 00000100:00000001:2.0:1787756943.554721:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756943.554724:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554726:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756943.554733:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756943/real 1787756943] req@ffff8ffb4790fb80 x1874588659598592/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756943.554745:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756943.554747:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756943.554751:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756943.554754:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756943.554756:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554758:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756943.554760:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756943.554765:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659598592/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756943.554774:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554777:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756943.554781:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554783:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756943.554785:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554787:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756943.554809:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756943.554813:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756943.554817:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756943.554820:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756943.554825:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598592:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756943.554829:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756943.554830:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756943.554832:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756943.554835:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756943.554837:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756943.554840:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756943.554844:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756943.554848:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756943.554851:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756943.554853:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756943.554855:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756943.554859:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554866:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756943.554868:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:2.0:1787756943.554874:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756943.554890:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756943.554896:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756943.554899:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756943.554901:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554903:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756943.554906:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554911:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304670731 00000100:00000001:2.0:1787756943.554914:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304671425 00000100:00000001:2.0:1787756943.554917:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304670731 00000100:00000001:2.0:1787756943.554920:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304671425 00000100:00000001:2.0:1787756943.554923:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756943.554925:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.554928:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756943.554933:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756943/real 1787756943] req@ffff8ffb4790d180 x1874588659598720/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756943.554942:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756943.554943:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756943.554946:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756943.554948:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756943.554950:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554952:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756943.554954:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756943.554959:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659598720/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756943.554967:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554969:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756943.554972:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554974:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756943.554975:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.554977:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756943.554995:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756943.554998:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756943.555001:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756943.555004:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756943.555008:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756943.555012:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756943.555015:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756943.555017:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756943.555020:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756943.555022:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756943.555025:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756943.555028:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756943.555032:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756943.555033:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756943.555035:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756943.555037:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756943.555041:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.555045:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756943.555047:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:2.0:1787756943.555052:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756943.555063:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756943.555067:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756943.555069:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756943.555071:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.555073:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756943.555076:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.555080:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304670731 00000100:00000001:2.0:1787756943.555083:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304671425 00000100:00000001:2.0:1787756943.555086:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304670731 00000100:00000001:2.0:1787756943.555089:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304671425 00000100:00000001:2.0:1787756943.555092:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756943.555094:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.555096:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756943.555101:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756943/real 1787756943] req@ffff8ffb4790e680 x1874588659598848/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756943.555109:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756943.555111:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756943.555113:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756943.555115:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756943.555116:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555119:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756943.555121:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756943.555126:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659598848/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756943.555135:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555137:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756943.555140:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555142:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756943.555144:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555145:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756943.555164:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756943.555167:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756943.555170:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756943.555173:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756943.555176:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659598848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756943.555180:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555247:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756943.555249:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756943.555251:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756943.555255:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756943.555257:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756943.555259:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756943.555262:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756943.555264:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756943.555267:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756943.555268:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555271:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756943.555272:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756943.555274:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756943.555276:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756943.555277:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756943.555278:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756943.555281:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756943.555282:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756943.555284:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756943.555287:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555289:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756943.555291:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756943.555293:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756943.555295:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756943.555296:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756943.555298:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756943.555300:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756943.555302:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756943.555303:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756943.555305:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555307:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756943.555312:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756943.555314:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756943.555316:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756943.555318:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756945.345524:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756945.345533:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756945.345539:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756946.369374:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756946.369378:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756946.369382:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756949.441451:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756949.441458:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756949.441464:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756950.465508:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756950.465514:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756950.465520:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756953.537465:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756953.537472:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756953.537478:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756954.561456:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756954.561462:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756954.561468:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756957.633434:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756957.633441:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756957.633447:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756958.657523:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756958.657530:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756958.657535:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756958.913379:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756958.913395:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756958.913400:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756958.913406:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756958.913413:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756958.913425:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10013 00000100:00080000:3.0:1787756958.913444:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756958.913449:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756958.913452:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756958.913458:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756958.913462:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756958.913467:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756958.913471:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756958.913474:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756958.913476:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756958.913483:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756958.913486:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756958.913490:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756958.913495:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756958.913500:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913503:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756958.913506:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756958.913527:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756958.913529:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913534:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756958.913539:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756958.913546:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913567:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756958.913581:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659600000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756958.913597:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756958.913603:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913625:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756958.913630:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756958.913635:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756958.913637:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756958.913639:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756958.913643:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756958.913646:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10013 00000100:00080000:3.0:1787756958.913651:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756958.913654:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756958.913656:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756958.913659:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756958.913661:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756958.913663:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756958.913666:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756958.913669:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756958.913671:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756958.913675:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756958.913678:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756958.913680:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756958.913687:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756958.913689:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913692:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756958.913698:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756958.913702:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913706:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756958.913711:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756958.913713:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913716:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756958.913719:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756958.913721:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913727:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756958.913734:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659600128/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756958.913752:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756958.913757:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913766:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756958.913774:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756958.913782:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756958.913788:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756958.913795:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756958.913801:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756958.913812:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10013 00000100:00080000:3.0:1787756958.913822:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756958.913828:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756958.913834:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756958.913840:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756958.913846:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756958.913849:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756958.913852:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756958.913854:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756958.913855:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756958.913859:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756958.913864:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756958.913868:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756958.913873:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756958.913880:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913888:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756958.913891:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756958.913893:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913896:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756958.913903:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756958.913907:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913910:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756958.913912:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756958.913913:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.913961:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756958.913970:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659600256/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756958.913993:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756958.913999:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.914002:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756958.914006:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756958.914011:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756958.914014:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756958.914016:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756958.914020:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756958.914024:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10013 00000100:00080000:3.0:1787756958.914030:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756958.914032:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756958.914034:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756958.914036:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756958.914037:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756958.914041:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756958.914044:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756958.914048:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756958.914051:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756958.914055:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756958.914059:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756958.914061:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756958.914066:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756958.914076:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.914085:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756958.914088:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756958.914093:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.914096:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756958.914105:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756958.914113:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.914117:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756958.914120:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756958.914127:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.914136:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756958.914148:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659600384/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756958.914169:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756958.914173:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756958.914178:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756961.729535:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756961.729541:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756961.729546:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756962.753434:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756962.753441:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756962.753446:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756964.033708:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756964.033720:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756964.033727:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756964.033734:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756964.033744:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756964.033758:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756964.033770:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756964.033787:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756964.033811:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659600512:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756964.033853:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756964.033866:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756964.033875:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756964.033888:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756964.033905:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.033942:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756964.033950:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:2.0:1787756964.033966:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756964.034053:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756964.034079:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756964.034090:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756964.034100:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034107:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756964.034120:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034130:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304691905 00000100:00000001:2.0:1787756964.034154:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756964.034170:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304691905 00000100:00000001:2.0:1787756964.034229:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756964.034237:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756964.034241:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034248:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756964.034265:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756964/real 1787756964] req@ffff8ffb4790d880 x1874588659600512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756964.034300:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756964.034306:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756964.034316:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756964.034326:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756964.034332:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034336:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756964.034344:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756964.034356:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659600512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756964.034436:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034443:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756964.034448:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034450:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756964.034451:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034455:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756964.034459:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756964.034463:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756964.034469:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756964.034473:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756964.034478:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659600512:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756964.034484:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034487:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756964.034489:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756964.034493:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756964.034499:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756964.034502:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756964.034505:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756964.034512:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756964.034515:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756964.034519:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756964.034520:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034524:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034534:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756964.034538:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034541:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756964.034543:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756964.034546:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756964.034548:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756964.034550:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756964.034553:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756964.034556:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756964.034559:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756964.034564:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659600640:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756964.034568:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756964.034571:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756964.034573:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756964.034576:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756964.034580:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034587:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756964.034589:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:2.0:1787756964.034596:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756964.034612:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756964.034618:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756964.034621:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756964.034623:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034625:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756964.034629:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034634:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304691214 00000100:00000001:2.0:1787756964.034639:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304691905 00000100:00000001:2.0:1787756964.034642:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304691214 00000100:00000001:2.0:1787756964.034645:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304691905 00000100:00000001:2.0:1787756964.034648:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756964.034651:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034653:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756964.034658:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756964/real 1787756964] req@ffff8ffb4790f100 x1874588659600640/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756964.034667:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756964.034669:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756964.034672:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756964.034674:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756964.034677:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034679:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756964.034681:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756964.034685:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659600640/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756964.034703:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034706:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756964.034709:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034711:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756964.034713:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034715:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756964.034737:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756964.034740:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756964.034744:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756964.034746:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756964.034750:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659600640:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756964.034755:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756964.034757:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756964.034759:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756964.034762:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756964.034764:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756964.034767:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756964.034770:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659600768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756964.034775:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756964.034777:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756964.034779:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756964.034781:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756964.034785:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034791:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756964.034793:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:2.0:1787756964.034798:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756964.034809:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756964.034814:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756964.034817:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756964.034819:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034822:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756964.034849:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034854:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304691211 00000100:00000001:2.0:1787756964.034858:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304691905 00000100:00000001:2.0:1787756964.034861:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304691211 00000100:00000001:2.0:1787756964.034864:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304691905 00000100:00000001:2.0:1787756964.034867:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756964.034869:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034871:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756964.034876:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756964/real 1787756964] req@ffff8ffb4790c380 x1874588659600768/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756964.034886:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756964.034887:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756964.034891:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756964.034893:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756964.034895:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034897:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756964.034899:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756964.034904:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659600768/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756964.034912:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034915:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756964.034918:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034920:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756964.034921:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.034923:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756964.034942:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756964.034944:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756964.034948:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756964.034950:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756964.034955:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659600768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756964.034960:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756964.034962:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756964.034964:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756964.034967:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756964.034969:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756964.034971:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756964.034975:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659600896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756964.034978:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756964.034980:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756964.034982:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756964.034985:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756964.034989:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.034994:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756964.034996:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:2.0:1787756964.035001:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756964.035015:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756964.035020:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756964.035023:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756964.035024:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.035027:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756964.035030:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.035034:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304691211 00000100:00000001:2.0:1787756964.035038:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304691905 00000100:00000001:2.0:1787756964.035041:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304691211 00000100:00000001:2.0:1787756964.035044:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304691905 00000100:00000001:2.0:1787756964.035046:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756964.035048:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.035051:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756964.035055:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756964/real 1787756964] req@ffff8ffb4790ce00 x1874588659600896/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756964.035073:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756964.035074:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756964.035077:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756964.035079:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756964.035081:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035083:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756964.035085:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756964.035089:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659600896/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756964.035097:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035100:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756964.035103:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035105:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756964.035107:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035109:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756964.035129:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756964.035132:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756964.035135:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756964.035138:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756964.035142:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659600896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756964.035147:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035149:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756964.035151:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756964.035153:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756964.035156:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756964.035157:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756964.035159:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756964.035163:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756964.035164:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756964.035167:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756964.035168:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035171:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756964.035173:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756964.035175:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756964.035177:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756964.035178:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756964.035205:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756964.035209:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756964.035211:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756964.035214:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756964.035215:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035218:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756964.035219:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756964.035221:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756964.035225:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756964.035227:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756964.035228:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756964.035230:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756964.035231:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756964.035233:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756964.035235:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035237:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756964.035242:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756964.035244:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756964.035246:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756964.035249:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756965.825507:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756965.825514:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756965.825519:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756966.849477:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756966.849484:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756966.849491:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787756969.153725:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756969.153777:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756969.153785:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756969.153794:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756969.153801:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756969.153820:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756969.153845:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756969.153862:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756969.153894:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659601024:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756969.153919:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756969.153930:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756969.153938:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756969.153959:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756969.153976:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154003:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756969.154008:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787756969.154017:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756969.154079:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756969.154101:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756969.154112:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756969.154119:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154140:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756969.154152:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154241:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304697026 00000100:00000001:2.0:1787756969.154270:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787756969.154286:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304697026 00000100:00000001:2.0:1787756969.154307:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787756969.154331:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756969.154341:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154348:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756969.154368:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756969/real 1787756969] req@ffff8ffb4a534a80 x1874588659601024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756969.154395:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756969.154399:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756969.154407:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756969.154412:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756969.154418:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154428:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756969.154435:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756969.154454:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659601024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756969.154481:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154487:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756969.154492:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154495:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756969.154498:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154504:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756969.154509:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756969.154514:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756969.154520:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756969.154523:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756969.154528:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659601024:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787756969.154535:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154538:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756969.154540:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756969.154543:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756969.154549:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756969.154553:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756969.154555:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756969.154562:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756969.154564:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756969.154568:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756969.154570:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154573:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154581:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756969.154583:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154586:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756969.154589:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787756969.154591:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756969.154594:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756969.154596:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756969.154599:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756969.154602:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756969.154605:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756969.154609:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659601152:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756969.154614:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756969.154616:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756969.154619:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756969.154621:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756969.154625:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154633:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756969.154635:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787756969.154640:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756969.154658:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756969.154664:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756969.154666:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756969.154668:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154671:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756969.154674:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154679:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304696334 00000100:00000001:2.0:1787756969.154683:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304697026 00000100:00000001:2.0:1787756969.154686:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304696334 00000100:00000001:2.0:1787756969.154689:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304697026 00000100:00000001:2.0:1787756969.154692:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756969.154694:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154697:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756969.154702:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756969/real 1787756969] req@ffff8ffb4a535c00 x1874588659601152/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756969.154712:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756969.154714:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756969.154716:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756969.154718:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756969.154720:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154722:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756969.154724:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756969.154729:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659601152/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756969.154761:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154765:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756969.154768:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154770:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756969.154771:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154775:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756969.154792:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756969.154795:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756969.154800:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756969.154803:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756969.154807:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659601152:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787756969.154812:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756969.154815:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756969.154817:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756969.154820:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756969.154822:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756969.154825:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756969.154829:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659601280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756969.154833:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756969.154835:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756969.154838:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756969.154840:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756969.154844:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154850:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756969.154852:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787756969.154857:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756969.154870:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756969.154876:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756969.154878:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756969.154880:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154882:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756969.154885:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154890:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304696331 00000100:00000001:2.0:1787756969.154893:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304697026 00000100:00000001:2.0:1787756969.154896:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304696331 00000100:00000001:2.0:1787756969.154899:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304697026 00000100:00000001:2.0:1787756969.154904:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756969.154906:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.154908:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756969.154913:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756969/real 1787756969] req@ffff8ffb4a534700 x1874588659601280/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756969.154923:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756969.154925:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756969.154928:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756969.154930:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756969.154932:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154934:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756969.154935:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756969.154940:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659601280/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756969.154948:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154951:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756969.154954:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154956:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756969.154957:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.154959:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756969.154973:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756969.154976:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756969.154979:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756969.154981:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756969.154985:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659601280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756969.154990:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787756969.154992:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787756969.154994:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756969.154997:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787756969.154999:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756969.155001:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756969.155006:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659601408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756969.155010:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787756969.155012:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787756969.155014:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756969.155017:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787756969.155020:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.155024:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787756969.155026:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787756969.155030:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787756969.155041:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787756969.155046:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787756969.155048:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756969.155050:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.155052:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787756969.155055:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.155059:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304696331 00000100:00000001:2.0:1787756969.155062:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304697026 00000100:00000001:2.0:1787756969.155064:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304696331 00000100:00000001:2.0:1787756969.155067:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304697026 00000100:00000001:2.0:1787756969.155070:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787756969.155072:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.155074:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787756969.155079:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756969/real 1787756969] req@ffff8ffb4a537800 x1874588659601408/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787756969.155087:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787756969.155089:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787756969.155092:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787756969.155093:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787756969.155095:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155097:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756969.155099:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787756969.155104:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659601408/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787756969.155120:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155123:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787756969.155126:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155128:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787756969.155130:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155132:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787756969.155146:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787756969.155148:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756969.155152:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787756969.155154:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787756969.155158:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659601408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787756969.155162:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155165:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756969.155166:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756969.155168:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756969.155171:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756969.155173:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756969.155174:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756969.155177:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756969.155179:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756969.155214:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756969.155217:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155219:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756969.155221:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756969.155223:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756969.155225:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756969.155227:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756969.155228:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756969.155231:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756969.155232:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756969.155234:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756969.155236:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155238:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787756969.155239:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787756969.155242:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787756969.155243:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787756969.155245:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787756969.155247:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787756969.155249:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787756969.155250:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787756969.155252:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787756969.155254:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155256:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787756969.155260:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787756969.155262:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756969.155264:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787756969.155266:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756969.921490:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756969.921497:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756969.921502:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756970.945471:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756970.945477:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756970.945483:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756974.017501:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756974.017511:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756974.017519:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756974.273421:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756974.273437:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756974.273441:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756974.273446:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756974.273453:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756974.273482:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10029 00000100:00080000:2.0:1787756974.273501:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756974.273506:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756974.273509:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756974.273516:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756974.273520:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756974.273525:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756974.273528:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756974.273531:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756974.273533:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756974.273539:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756974.273542:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756974.273546:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756974.273552:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756974.273557:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273560:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756974.273564:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756974.273583:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756974.273586:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273591:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756974.273595:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756974.273602:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273622:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756974.273636:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659601536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756974.273687:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756974.273694:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273719:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756974.273724:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756974.273729:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756974.273731:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756974.273734:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756974.273738:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756974.273741:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10029 00000100:00080000:2.0:1787756974.273746:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756974.273750:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756974.273752:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756974.273755:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756974.273757:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756974.273759:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756974.273762:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756974.273765:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756974.273767:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756974.273772:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756974.273777:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756974.273781:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756974.273789:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756974.273793:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273796:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756974.273803:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756974.273812:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273817:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756974.273825:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756974.273831:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273837:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756974.273843:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756974.273847:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273860:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756974.273875:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659601664/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756974.273906:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756974.273911:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.273921:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756974.273930:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756974.273937:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756974.273946:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756974.273949:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756974.273953:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756974.273959:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10029 00000100:00080000:2.0:1787756974.273963:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756974.273966:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756974.273968:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756974.273972:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756974.273978:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756974.273984:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756974.273990:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756974.273996:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756974.274000:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756974.274005:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756974.274008:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756974.274012:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756974.274017:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756974.274020:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274022:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756974.274025:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756974.274028:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274030:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756974.274037:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756974.274039:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274041:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756974.274043:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756974.274046:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274055:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756974.274066:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659601792/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756974.274080:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756974.274087:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274092:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756974.274101:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756974.274112:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756974.274117:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756974.274120:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756974.274124:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756974.274133:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10029 00000100:00080000:2.0:1787756974.274138:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756974.274143:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756974.274145:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756974.274148:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756974.274151:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756974.274155:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756974.274163:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756974.274166:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756974.274169:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756974.274220:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756974.274224:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756974.274226:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756974.274231:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756974.274234:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274236:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756974.274239:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756974.274243:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274254:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756974.274260:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756974.274262:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274264:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756974.274266:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756974.274270:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274280:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756974.274287:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659601920/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756974.274308:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756974.274313:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756974.274320:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756975.041478:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756975.041499:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756975.041504:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756978.113530:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756978.113539:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756978.113545:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756979.137621:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756979.137630:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756979.137636:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756979.393328:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756979.393342:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756979.393345:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756979.393350:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756979.393355:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756979.393364:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10034 00000100:00080000:2.0:1787756979.393377:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756979.393380:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756979.393382:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756979.393388:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756979.393390:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756979.393394:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756979.393397:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756979.393399:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756979.393401:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756979.393406:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756979.393409:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756979.393412:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756979.393416:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756979.393420:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393422:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756979.393424:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756979.393437:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756979.393439:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393443:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756979.393457:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756979.393462:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393482:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756979.393494:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659602048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756979.393507:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756979.393523:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393542:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756979.393546:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756979.393553:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756979.393575:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756979.393579:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756979.393594:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756979.393598:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10034 00000100:00080000:2.0:1787756979.393603:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756979.393605:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756979.393608:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756979.393612:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756979.393614:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756979.393617:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756979.393619:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756979.393622:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756979.393624:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756979.393631:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756979.393635:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756979.393637:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756979.393643:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756979.393645:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393647:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756979.393654:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756979.393664:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393669:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756979.393678:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756979.393683:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393689:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756979.393693:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756979.393702:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393716:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756979.393733:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659602176/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756979.393786:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756979.393799:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393816:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756979.393823:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756979.393831:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756979.393833:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756979.393836:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756979.393841:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756979.393846:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10034 00000100:00080000:2.0:1787756979.393851:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756979.393861:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756979.393869:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756979.393892:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756979.393895:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756979.393905:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756979.393911:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756979.393916:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756979.393920:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756979.393933:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756979.393942:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756979.393947:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756979.393958:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756979.393972:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.393995:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756979.394006:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756979.394031:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394036:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756979.394045:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756979.394049:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394054:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756979.394058:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756979.394069:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394092:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756979.394138:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659602304/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756979.394188:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756979.394192:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394196:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756979.394245:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756979.394260:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756979.394268:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756979.394272:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756979.394281:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756979.394297:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10034 00000100:00080000:2.0:1787756979.394320:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756979.394328:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756979.394343:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756979.394361:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756979.394373:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756979.394382:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756979.394396:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756979.394413:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756979.394420:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756979.394441:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756979.394454:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756979.394466:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756979.394474:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756979.394483:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394487:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756979.394495:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756979.394511:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394516:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756979.394523:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756979.394528:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394534:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756979.394546:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756979.394578:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394608:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756979.394628:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659602432/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756979.394667:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756979.394675:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756979.394704:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756980.161441:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756980.161452:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756980.161460:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756982.209499:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756982.209530:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756982.209549:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756983.233442:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756983.233449:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756983.233454:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756984.257558:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756984.257577:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756984.257585:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756984.513391:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756984.513408:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756984.513413:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756984.513418:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756984.513426:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756984.513437:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10039 00000100:00080000:3.0:1787756984.513456:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756984.513462:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756984.513490:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756984.513497:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756984.513501:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756984.513506:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756984.513511:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756984.513515:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756984.513519:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756984.513527:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756984.513532:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756984.513537:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756984.513543:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756984.513548:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.513552:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756984.513556:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756984.513571:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756984.513574:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.513579:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756984.513585:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756984.513599:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.513619:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756984.513635:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659602560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756984.513650:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756984.513657:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.513686:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756984.513693:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756984.513701:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756984.513705:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756984.513710:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756984.513717:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756984.513725:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10039 00000100:00080000:3.0:1787756984.513737:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756984.513742:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756984.513746:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756984.513751:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756984.513755:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756984.513760:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756984.513768:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756984.513780:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756984.513788:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756984.513808:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756984.513822:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756984.514309:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756984.514413:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756984.514423:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514428:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756984.514438:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756984.514445:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514453:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756984.514499:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756984.514504:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514510:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756984.514515:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756984.514521:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514536:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756984.514553:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659602688/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756984.514572:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756984.514577:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514598:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756984.514605:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756984.514613:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756984.514616:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756984.514620:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756984.514627:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756984.514636:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10039 00000100:00080000:3.0:1787756984.514648:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756984.514657:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756984.514662:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756984.514669:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756984.514673:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756984.514681:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756984.514687:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756984.514696:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756984.514704:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756984.514716:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756984.514727:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756984.514733:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756984.514742:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756984.514752:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514757:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756984.514764:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756984.514770:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514777:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756984.514794:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756984.514799:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514802:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756984.514805:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756984.514809:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514822:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756984.514839:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659602816/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756984.514860:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756984.514869:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.514879:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756984.514887:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756984.514899:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756984.514907:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756984.514914:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756984.514925:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756984.514935:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10039 00000100:00080000:3.0:1787756984.514949:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756984.514958:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756984.514964:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756984.514969:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756984.514974:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756984.514980:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756984.514990:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756984.514997:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756984.515004:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756984.515016:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756984.515024:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756984.515034:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756984.515042:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756984.515052:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.515057:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756984.515063:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756984.515073:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.515080:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756984.515092:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756984.515098:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.515105:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756984.515113:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756984.515122:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.515139:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756984.515155:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659602944/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756984.515255:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756984.515266:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756984.515279:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756986.305521:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756986.305527:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756986.305532:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756987.329601:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756987.329608:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756987.329614:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756988.353477:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756988.353482:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756988.353486:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756989.633303:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756989.633316:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756989.633320:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756989.633324:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756989.633330:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756989.633340:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10044 00000100:00080000:3.0:1787756989.633354:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756989.633358:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756989.633361:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756989.633366:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756989.633369:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756989.633396:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756989.633402:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756989.633406:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756989.633408:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756989.633415:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756989.633418:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756989.633422:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756989.633426:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756989.633430:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633432:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756989.633436:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756989.633450:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756989.633453:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633457:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756989.633461:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756989.633466:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633482:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756989.633516:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659603072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756989.633538:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756989.633546:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633574:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756989.633581:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756989.633586:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756989.633588:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756989.633590:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756989.633594:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756989.633598:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10044 00000100:00080000:3.0:1787756989.633604:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756989.633608:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756989.633609:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756989.633613:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756989.633615:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756989.633617:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756989.633620:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756989.633625:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756989.633627:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756989.633632:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756989.633636:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756989.633638:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756989.633645:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756989.633648:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633651:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756989.633658:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756989.633663:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633667:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756989.633685:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756989.633687:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633690:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756989.633692:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756989.633695:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633704:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756989.633717:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659603200/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756989.633734:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756989.633741:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633749:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756989.633757:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756989.633776:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756989.633780:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756989.633784:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756989.633796:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756989.633810:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10044 00000100:00080000:3.0:1787756989.633825:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756989.633839:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756989.633845:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756989.633863:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756989.633869:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756989.633879:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756989.633889:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756989.633894:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756989.633898:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756989.633909:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756989.633915:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756989.633918:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756989.633925:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756989.633953:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633957:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756989.633960:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756989.633965:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633968:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756989.633975:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756989.633976:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633979:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756989.633982:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756989.633987:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.633998:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756989.634012:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659603328/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756989.634025:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756989.634028:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.634032:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756989.634036:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756989.634043:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756989.634047:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756989.634050:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756989.634056:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756989.634062:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10044 00000100:00080000:3.0:1787756989.634072:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756989.634077:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756989.634079:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756989.634085:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756989.634088:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756989.634092:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756989.634096:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756989.634102:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756989.634105:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756989.634117:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756989.634134:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756989.634142:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756989.634147:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756989.634152:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.634156:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756989.634166:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756989.634171:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.634174:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756989.634228:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756989.634230:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.634233:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756989.634235:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756989.634238:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.634247:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756989.634257:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659603456/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756989.634277:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756989.634282:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756989.634305:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787756990.401509:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756990.401514:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756990.401519:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756991.425548:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756991.425554:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756991.425558:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756992.449434:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756992.449440:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756992.449444:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756994.497480:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756994.497486:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756994.497491:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756994.753417:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756994.753434:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787756994.753439:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756994.753444:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756994.753451:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756994.753462:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10049 00000100:00080000:3.0:1787756994.753481:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756994.753486:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756994.753489:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756994.753495:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756994.753500:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756994.753505:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787756994.753508:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787756994.753511:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756994.753513:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756994.753520:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756994.753524:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756994.753528:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756994.753534:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756994.753539:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753542:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787756994.753545:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756994.753567:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756994.753569:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753575:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756994.753579:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756994.753587:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753607:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756994.753621:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659603584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756994.753637:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756994.753643:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753671:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756994.753676:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756994.753681:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787756994.753683:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756994.753686:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756994.753689:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756994.753692:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10049 00000100:00080000:3.0:1787756994.753697:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756994.753699:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756994.753701:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756994.753705:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756994.753707:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756994.753710:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787756994.753713:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787756994.753716:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756994.753718:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756994.753725:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756994.753729:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756994.753733:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756994.753742:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756994.753745:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753748:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756994.753757:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756994.753765:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753778:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756994.753794:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756994.753801:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753807:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756994.753811:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756994.753823:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753848:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756994.753863:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659603712/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756994.753903:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756994.753920:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.753937:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756994.753947:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756994.753957:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787756994.753963:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756994.753976:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756994.753989:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756994.753998:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10049 00000100:00080000:3.0:1787756994.754008:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756994.754012:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756994.754015:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756994.754019:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756994.754021:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756994.754025:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787756994.754029:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787756994.754032:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756994.754035:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756994.754041:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756994.754045:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756994.754048:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756994.754054:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756994.754060:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754078:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756994.754094:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756994.754107:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754118:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756994.754135:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756994.754146:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754167:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756994.754178:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756994.754243:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754264:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756994.754312:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659603840/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756994.754373:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756994.754391:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754415:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787756994.754424:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756994.754436:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787756994.754444:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787756994.754459:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787756994.754465:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787756994.754473:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10049 00000100:00080000:3.0:1787756994.754482:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787756994.754486:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756994.754489:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787756994.754493:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787756994.754499:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787756994.754503:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787756994.754511:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787756994.754523:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787756994.754532:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787756994.754557:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787756994.754568:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756994.754575:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787756994.754588:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787756994.754597:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754610:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787756994.754614:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787756994.754623:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754636:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787756994.754647:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787756994.754655:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754661:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787756994.754674:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787756994.754680:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754695:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787756994.754718:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659603968/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787756994.754780:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787756994.754790:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787756994.754803:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787756995.521539:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756995.521545:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756995.521551:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756996.545457:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756996.545464:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756996.545470:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787756998.593540:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787756998.593549:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787756998.593557:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787756999.617515:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787756999.617521:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787756999.617526:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756999.873412:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756999.873430:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787756999.873434:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756999.873439:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756999.873447:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756999.873459:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10054 00000100:00080000:2.0:1787756999.873479:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756999.873484:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756999.873487:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756999.873494:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756999.873498:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756999.873502:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787756999.873506:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787756999.873509:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756999.873511:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756999.873517:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756999.873521:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756999.873525:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756999.873530:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756999.873535:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873538:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787756999.873541:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756999.873556:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756999.873558:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873563:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756999.873568:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756999.873575:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873605:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756999.873624:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659604096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756999.873647:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756999.873656:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873695:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756999.873703:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756999.873711:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787756999.873715:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756999.873718:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756999.873725:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756999.873731:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10054 00000100:00080000:2.0:1787756999.873739:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756999.873744:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756999.873748:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756999.873753:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756999.873755:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756999.873760:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787756999.873764:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787756999.873768:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756999.873770:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756999.873777:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756999.873783:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756999.873787:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756999.873797:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756999.873801:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873805:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756999.873813:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756999.873819:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873823:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756999.873832:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756999.873835:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873839:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756999.873842:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756999.873846:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873858:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756999.873871:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659604224/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756999.873888:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756999.873894:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.873899:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756999.873906:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756999.873921:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787756999.873929:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756999.873938:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756999.873951:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756999.873963:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10054 00000100:00080000:2.0:1787756999.873976:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756999.873988:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756999.873996:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756999.874005:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756999.874010:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756999.874015:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787756999.874026:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787756999.874035:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756999.874043:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756999.874053:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756999.874063:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756999.874071:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756999.874082:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756999.874088:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874097:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756999.874103:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756999.874109:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874112:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756999.874122:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756999.874127:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874136:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756999.874142:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756999.874149:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874162:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756999.874178:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659604352/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756999.874287:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756999.874293:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874304:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787756999.874311:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756999.874317:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787756999.874321:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787756999.874325:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787756999.874332:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787756999.874339:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10054 00000100:00080000:2.0:1787756999.874349:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787756999.874354:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756999.874358:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787756999.874363:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787756999.874366:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787756999.874371:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787756999.874376:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787756999.874380:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787756999.874383:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787756999.874393:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787756999.874406:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756999.874416:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787756999.874425:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787756999.874430:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874436:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787756999.874447:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787756999.874454:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874463:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787756999.874481:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787756999.874488:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874492:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787756999.874497:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787756999.874506:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874522:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787756999.874539:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659604480/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787756999.874568:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787756999.874575:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787756999.874589:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757000.641435:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757000.641442:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757000.641448:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757002.689460:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757002.689467:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757002.689472:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757003.713433:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757003.713438:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757003.713442:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757004.737523:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757004.737532:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757004.737538:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757004.993310:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757004.993326:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757004.993330:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757004.993335:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757004.993343:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757004.993354:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10059 00000100:00080000:2.0:1787757004.993370:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757004.993375:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757004.993379:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757004.993385:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757004.993389:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757004.993394:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757004.993398:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757004.993401:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757004.993403:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757004.993409:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757004.993413:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757004.993417:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757004.993421:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757004.993426:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993429:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757004.993432:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757004.993449:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757004.993451:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993456:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757004.993462:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757004.993469:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993489:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757004.993504:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659604608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757004.993519:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757004.993525:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993548:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757004.993553:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757004.993558:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757004.993560:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757004.993562:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757004.993566:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757004.993569:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10059 00000100:00080000:2.0:1787757004.993574:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757004.993577:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757004.993580:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757004.993584:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757004.993586:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757004.993589:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757004.993593:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757004.993596:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757004.993598:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757004.993602:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757004.993605:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757004.993607:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757004.993617:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757004.993627:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993632:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757004.993642:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757004.993652:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993677:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757004.993685:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757004.993688:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993697:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757004.993702:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757004.993714:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993733:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757004.993760:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659604736/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757004.993789:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757004.993795:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.993805:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757004.993813:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757004.993826:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757004.993834:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757004.993840:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757004.993851:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757004.993859:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10059 00000100:00080000:2.0:1787757004.993871:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757004.993876:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757004.993880:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757004.993886:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757004.993890:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757004.993894:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757004.993900:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757004.993905:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757004.993942:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757004.993966:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757004.993983:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757004.993991:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757004.994001:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757004.994013:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994024:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757004.994033:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757004.994041:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994046:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757004.994068:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757004.994076:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994090:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757004.994130:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757004.994140:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994161:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757004.994236:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659604864/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757004.994270:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757004.994284:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994295:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757004.994311:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757004.994332:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757004.994341:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757004.994350:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757004.994358:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757004.994368:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10059 00000100:00080000:2.0:1787757004.994377:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757004.994383:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757004.994389:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757004.994395:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757004.994402:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757004.994409:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757004.994414:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757004.994426:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757004.994431:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757004.994440:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757004.994448:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757004.994454:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757004.994463:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757004.994470:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994490:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757004.994497:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757004.994514:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994526:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757004.994542:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757004.994551:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994558:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757004.994564:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757004.994573:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994590:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757004.994612:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659604992/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757004.994653:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757004.994663:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757004.994678:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757006.785474:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757006.785483:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757006.785491:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757007.809490:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757007.809499:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757007.809505:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757008.833553:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757008.833562:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757008.833568:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757010.113409:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757010.113426:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757010.113431:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757010.113436:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757010.113443:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757010.113455:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10064 00000100:00080000:3.0:1787757010.113471:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757010.113489:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757010.113493:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757010.113507:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757010.113512:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757010.113518:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757010.113523:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757010.113540:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757010.113543:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757010.113558:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757010.113563:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757010.113569:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757010.113574:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757010.113581:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113600:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757010.113604:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757010.113622:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757010.113625:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113630:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757010.113634:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757010.113642:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113660:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757010.113679:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659605120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757010.113698:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757010.113705:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113733:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757010.113740:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757010.113746:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757010.113749:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757010.113752:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757010.113758:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757010.113764:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10064 00000100:00080000:3.0:1787757010.113773:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757010.113777:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757010.113781:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757010.113786:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757010.113788:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757010.113792:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757010.113799:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757010.113807:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757010.113812:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757010.113821:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757010.113830:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757010.113835:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757010.113849:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757010.113858:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113862:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757010.113867:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757010.113871:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113874:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757010.113883:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757010.113886:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113890:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757010.113893:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757010.113897:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113908:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757010.113921:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659605248/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757010.113939:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757010.113944:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.113950:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757010.113955:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757010.113964:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757010.113966:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757010.113968:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757010.113974:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757010.113979:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10064 00000100:00080000:3.0:1787757010.113988:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757010.113994:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757010.114006:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757010.114040:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757010.114043:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757010.114048:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757010.114053:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757010.114056:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757010.114059:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757010.114069:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757010.114074:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757010.114078:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757010.114084:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757010.114088:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114092:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757010.114096:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757010.114103:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114112:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757010.114124:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757010.114129:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114137:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757010.114143:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757010.114151:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114167:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757010.114224:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659605376/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757010.114241:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757010.114246:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114253:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757010.114265:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757010.114280:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757010.114286:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757010.114292:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757010.114301:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757010.114310:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10064 00000100:00080000:3.0:1787757010.114320:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757010.114329:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757010.114333:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757010.114340:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757010.114344:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757010.114350:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757010.114355:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757010.114361:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757010.114373:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757010.114389:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757010.114399:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757010.114407:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757010.114415:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757010.114422:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114426:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757010.114428:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757010.114431:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114433:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757010.114439:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757010.114442:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114450:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757010.114454:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757010.114458:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114468:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757010.114488:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659605504/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757010.114519:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757010.114529:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757010.114540:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757010.881464:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757010.881485:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757010.881492:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757011.905464:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757011.905473:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757011.905480:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757012.929461:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757012.929468:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757012.929486:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757014.977496:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757014.977506:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757014.977513:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757015.233442:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757015.233455:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757015.233458:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757015.233463:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757015.233468:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757015.233478:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10069 00000100:00080000:2.0:1787757015.233492:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757015.233496:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757015.233500:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757015.233505:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757015.233508:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757015.233511:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757015.233515:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757015.233517:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757015.233519:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757015.233523:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757015.233526:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757015.233529:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757015.233534:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757015.233537:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233540:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757015.233543:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757015.233558:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757015.233560:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233564:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757015.233567:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757015.233572:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233598:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757015.233610:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659605632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757015.233623:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757015.233628:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233656:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757015.233661:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757015.233665:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757015.233667:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757015.233669:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757015.233672:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757015.233675:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10069 00000100:00080000:2.0:1787757015.233679:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757015.233684:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757015.233686:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757015.233689:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757015.233691:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757015.233694:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757015.233699:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757015.233704:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757015.233707:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757015.233717:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757015.233732:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757015.233741:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757015.233754:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757015.233767:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233773:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757015.233784:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757015.233797:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233805:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757015.233818:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757015.233825:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233833:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757015.233840:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757015.233849:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233866:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757015.233885:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659605760/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757015.233918:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757015.233940:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.233947:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757015.233951:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757015.233956:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757015.233958:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757015.233961:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757015.233965:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757015.233970:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10069 00000100:00080000:2.0:1787757015.233979:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757015.233982:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757015.233984:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757015.233987:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757015.233988:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757015.233991:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757015.234001:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757015.234008:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757015.234023:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757015.234033:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757015.234040:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757015.234052:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757015.234062:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757015.234075:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234080:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757015.234087:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757015.234098:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234107:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757015.234122:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757015.234129:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234137:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757015.234143:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757015.234151:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234176:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757015.234302:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659605888/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757015.234360:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757015.234370:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234386:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757015.234406:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757015.234426:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757015.234432:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757015.234438:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757015.234450:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757015.234465:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10069 00000100:00080000:2.0:1787757015.234479:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757015.234487:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757015.234491:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757015.234506:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757015.234520:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757015.234534:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757015.234538:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757015.234541:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757015.234544:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757015.234562:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757015.234568:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757015.234576:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757015.234584:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757015.234592:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234596:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757015.234602:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757015.234614:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234628:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757015.234636:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757015.234640:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234652:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757015.234658:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757015.234664:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234676:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757015.234699:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659606016/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757015.234760:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757015.234764:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757015.234777:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757016.001521:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757016.001552:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757016.001560:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757017.025456:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757017.025480:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757017.025488:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757019.073435:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757019.073444:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757019.073452:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757020.097543:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757020.097555:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757020.097564:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787757020.353732:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757020.353744:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757020.353752:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757020.353760:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757020.353767:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757020.353780:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757020.353791:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757020.353800:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757020.353821:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659606144:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757020.353988:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757020.354301:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757020.354308:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757020.354313:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757020.354321:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354349:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757020.354353:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133078272 : -123165576473344 : ffff8ffb4790d500) 00000100:00000001:3.0:1787757020.354361:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757020.354512:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757020.354530:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757020.354536:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757020.354540:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354543:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757020.354551:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354557:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304748226 00000100:00000001:3.0:1787757020.354572:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757020.354581:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304748226 00000100:00000001:3.0:1787757020.354586:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757020.354591:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757020.354595:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354600:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757020.354611:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757020/real 1787757020] req@ffff8ffb4790d500 x1874588659606144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757020.354635:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757020.354638:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757020.354647:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757020.354652:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757020.354655:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.354661:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757020.354665:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757020.354676:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d500 x1874588659606144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757020.354694:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.354701:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757020.354708:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.354712:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757020.354714:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.354719:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757020.354722:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757020.354727:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757020.354733:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757020.354737:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757020.354743:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659606144:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757020.354751:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.354754:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757020.354756:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757020.354760:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757020.354766:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757020.354769:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757020.354773:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757020.354779:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757020.354781:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757020.354784:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757020.354786:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.354789:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.354800:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757020.354803:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354806:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757020.354808:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757020.354811:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757020.354813:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757020.354815:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757020.354818:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757020.354821:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757020.354824:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757020.354829:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606272:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757020.354860:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757020.354863:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757020.354866:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757020.354868:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757020.354873:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354882:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757020.354884:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133087232 : -123165576464384 : ffff8ffb4790f800) 00000100:00000001:3.0:1787757020.354891:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757020.354911:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757020.354918:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757020.354921:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757020.354924:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354926:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757020.354930:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354936:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304747534 00000100:00000001:3.0:1787757020.354942:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304748226 00000100:00000001:3.0:1787757020.354947:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304747534 00000100:00000001:3.0:1787757020.354951:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304748226 00000100:00000001:3.0:1787757020.354957:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757020.354960:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.354964:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757020.354973:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757020/real 1787757020] req@ffff8ffb4790f800 x1874588659606272/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757020.354991:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757020.354994:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757020.354999:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757020.355003:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757020.355005:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355010:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757020.355013:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757020.355022:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f800 x1874588659606272/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757020.355037:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355042:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757020.355048:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355050:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757020.355052:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355055:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757020.355078:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757020.355082:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757020.355087:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757020.355089:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757020.355094:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606272:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757020.355100:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757020.355102:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757020.355104:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757020.355107:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757020.355110:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757020.355113:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757020.355118:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606400:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757020.355122:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757020.355125:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757020.355129:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757020.355132:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757020.355138:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355150:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757020.355153:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133081856 : -123165576469760 : ffff8ffb4790e300) 00000100:00000001:3.0:1787757020.355160:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757020.355237:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757020.355248:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757020.355252:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757020.355255:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355258:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757020.355263:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355268:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304747532 00000100:00000001:3.0:1787757020.355272:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304748226 00000100:00000001:3.0:1787757020.355275:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304747532 00000100:00000001:3.0:1787757020.355279:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304748226 00000100:00000001:3.0:1787757020.355281:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757020.355284:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355286:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757020.355292:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757020/real 1787757020] req@ffff8ffb4790e300 x1874588659606400/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757020.355303:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757020.355304:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757020.355308:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757020.355310:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757020.355312:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355314:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757020.355317:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757020.355322:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e300 x1874588659606400/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757020.355340:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355343:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757020.355347:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355349:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757020.355350:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355353:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757020.355372:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757020.355375:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757020.355379:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757020.355381:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757020.355385:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606400:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757020.355390:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757020.355392:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757020.355394:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757020.355397:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757020.355400:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757020.355402:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757020.355406:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757020.355410:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757020.355412:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757020.355415:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757020.355417:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757020.355421:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355427:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757020.355429:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:3.0:1787757020.355434:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757020.355450:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757020.355456:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757020.355459:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757020.355461:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355463:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757020.355466:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355470:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304747532 00000100:00000001:3.0:1787757020.355474:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304748226 00000100:00000001:3.0:1787757020.355477:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304747532 00000100:00000001:3.0:1787757020.355480:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304748226 00000100:00000001:3.0:1787757020.355483:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757020.355485:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355487:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757020.355492:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757020/real 1787757020] req@ffff8ffb4790ce00 x1874588659606528/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757020.355501:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757020.355502:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757020.355505:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757020.355507:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757020.355509:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355511:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757020.355513:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757020.355518:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659606528/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757020.355526:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355528:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757020.355531:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355533:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757020.355535:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355536:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757020.355556:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757020.355559:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757020.355563:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757020.355566:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757020.355571:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757020.355576:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355578:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757020.355580:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757020.355583:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757020.355585:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757020.355587:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757020.355588:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757020.355593:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757020.355595:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757020.355598:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757020.355600:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355603:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757020.355605:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757020.355607:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757020.355610:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757020.355613:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757020.355615:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757020.355618:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757020.355621:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757020.355624:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757020.355627:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355630:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757020.355632:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757020.355635:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757020.355639:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757020.355641:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757020.355644:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757020.355648:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757020.355650:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757020.355654:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757020.355657:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355660:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757020.355668:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757020.355671:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757020.355675:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757020.355679:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757021.121471:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757021.121478:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757021.121497:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757023.169484:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757023.169491:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757023.169496:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757024.193357:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757024.193361:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757024.193364:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757025.217549:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757025.217557:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757025.217563:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757025.473396:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757025.473412:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757025.473416:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757025.473421:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757025.473428:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757025.473438:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10080 00000100:00080000:2.0:1787757025.473465:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757025.473470:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757025.473473:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757025.473479:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757025.473483:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757025.473488:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757025.473492:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757025.473495:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757025.473497:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757025.473503:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757025.473507:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757025.473510:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757025.473516:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757025.473520:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.473523:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757025.473526:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757025.473546:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757025.473549:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.473569:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757025.473574:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757025.473581:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.473611:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757025.473627:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659606656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757025.473643:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757025.473650:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.473683:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757025.473687:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757025.473693:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757025.473695:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757025.473697:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757025.473701:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757025.473704:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10080 00000100:00080000:2.0:1787757025.473709:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757025.473711:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757025.473714:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757025.473717:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757025.473720:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757025.473723:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757025.473727:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757025.473731:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757025.473732:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757025.473737:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757025.473740:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757025.473769:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757025.473789:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757025.473792:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.473794:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757025.473799:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:2.0:1787757025.473801:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757025.473809:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:2.0:1787757025.473810:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.473817:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757025.473818:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.473825:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787757025.473829:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757025.473831:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757025.473837:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.473843:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757025.473844:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757025.473851:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757025.473852:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757025.473855:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.473859:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.473867:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757025.473873:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659606656:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1787757025.473883:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659606784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757025.473887:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757025.473896:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787757025.473903:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:3.0:1787757025.473904:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.473912:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757025.473913:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.473922:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.473925:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757025.473927:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757025.473937:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757025.473939:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757025.473942:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757025.473946:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757025.473950:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10080 00000100:00000001:3.0:1787757025.473956:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787757025.473957:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757025.473962:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:2.0:1787757025.473965:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757025.473972:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757025.473975:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787757025.473976:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757025.473982:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757025.473987:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757025.473990:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757025.473993:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757025.473995:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757025.473999:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757025.474002:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757025.474004:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757025.474008:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757025.474011:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474014:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757025.474018:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757025.474021:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474023:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757025.474032:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757025.474034:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474036:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757025.474038:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757025.474040:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474048:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757025.474058:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659606912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757025.474060:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757025.474075:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757025.474079:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474084:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757025.474089:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787757025.474091:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:2.0:1787757025.474094:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757025.474099:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757025.474102:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757025.474103:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757025.474106:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787757025.474109:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757025.474114:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10080 00000100:00000001:3.0:1787757025.474128:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:2.0:1787757025.474129:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757025.474136:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757025.474138:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757025.474141:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787757025.474143:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757025.474144:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757025.474153:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757025.474157:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304753345 00000020:00000001:2.0:1787757025.474160:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757025.474170:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757025.474173:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757025.474179:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:2.0:1787757025.474235:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757025.474247:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757025.474257:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757025.474257:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304753346 02000000:00000001:2.0:1787757025.474268:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757025.474277:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474277:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757025.474284:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757025.474290:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757025.474293:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00000001:2.0:1787757025.474295:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474305:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474307:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474315:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757025.474316:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757025.474322:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474328:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:3.0:1787757025.474329:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757025/real 1787757025] req@ffff8ffb47a39880 x1874588659606656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757025.474331:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757025.474338:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474351:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787757025.474358:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757025.474364:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:2.0:1787757025.474367:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659607040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757025.474375:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757025.474383:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757025.474396:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757025.474397:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757025.474407:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474407:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474421:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757025.474428:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757025.474443:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659606656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757025.474471:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474479:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757025.474484:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474487:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757025.474489:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474492:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757025.474495:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757025.474499:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757025.474507:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757025.474511:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757025.474516:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659606656:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757025.474524:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474528:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757025.474530:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757025.474534:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757025.474540:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757025.474544:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757025.474547:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757025.474555:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757025.474557:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757025.474562:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757025.474564:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474567:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474578:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757025.474581:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474585:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757025.474587:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757025.474589:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757025.474591:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757025.474593:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.474608:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757025.474611:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757025.474614:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757025.474618:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606784:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757025.474623:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757025.474626:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757025.474628:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.474630:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757025.474635:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474644:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757025.474646:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:3.0:1787757025.474653:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757025.474687:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757025.474698:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757025.474702:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757025.474705:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474708:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757025.474714:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474722:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304752654 00000100:00000001:3.0:1787757025.474728:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304753345 00000100:00000001:3.0:1787757025.474733:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304752654 00000100:00000001:3.0:1787757025.474738:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304753346 00000100:00000001:3.0:1787757025.474765:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757025.474771:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474786:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757025.474796:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757025/real 1787757025] req@ffff8ffb47a39500 x1874588659606784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757025.474813:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757025.474815:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757025.474820:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757025.474823:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757025.474825:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474827:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757025.474829:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757025.474835:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659606784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757025.474844:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474848:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757025.474851:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474853:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757025.474855:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.474857:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757025.474877:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757025.474881:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757025.474885:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757025.474887:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757025.474891:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606784:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757025.474897:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757025.474899:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757025.474901:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.474904:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757025.474907:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757025.474910:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757025.474913:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606912:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757025.474918:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757025.474921:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757025.474923:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.474925:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757025.474930:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474938:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757025.474940:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:3.0:1787757025.474946:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757025.474963:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757025.474971:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757025.474976:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757025.474978:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474982:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757025.474987:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.474994:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304752651 00000100:00000001:3.0:1787757025.474999:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304753345 00000100:00000001:3.0:1787757025.475004:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304752651 00000100:00000001:3.0:1787757025.475008:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304753346 00000100:00000001:3.0:1787757025.475013:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757025.475017:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.475021:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757025.475029:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757025/real 1787757025] req@ffff8ffb47a3a680 x1874588659606912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757025.475047:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757025.475050:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757025.475055:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757025.475058:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757025.475062:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475066:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757025.475068:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757025.475076:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659606912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757025.475094:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475097:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757025.475100:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475102:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757025.475104:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475106:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757025.475136:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757025.475139:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757025.475142:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757025.475145:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757025.475149:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659606912:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757025.475154:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757025.475156:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757025.475158:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.475161:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757025.475164:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757025.475166:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757025.475170:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757025.475174:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757025.475177:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757025.475179:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757025.475217:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757025.475225:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.475237:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757025.475240:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:3.0:1787757025.475248:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757025.475274:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757025.475284:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757025.475289:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757025.475292:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.475296:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757025.475302:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.475307:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304752652 00000100:00000001:3.0:1787757025.475311:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304753345 00000100:00000001:3.0:1787757025.475314:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304752652 00000100:00000001:3.0:1787757025.475317:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304753346 00000100:00000001:3.0:1787757025.475320:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757025.475323:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.475325:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757025.475331:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757025/real 1787757025] req@ffff8ffb47a38a80 x1874588659607040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757025.475342:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757025.475343:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757025.475347:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757025.475349:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757025.475351:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475353:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757025.475355:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757025.475360:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659607040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757025.475369:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475372:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757025.475376:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475378:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757025.475380:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475383:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757025.475416:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757025.475422:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757025.475429:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757025.475434:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757025.475442:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757025.475449:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475463:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757025.475466:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757025.475469:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757025.475474:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757025.475477:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757025.475480:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757025.475484:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757025.475486:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757025.475491:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757025.475493:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475495:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757025.475497:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757025.475499:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757025.475501:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757025.475503:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757025.475504:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757025.475508:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757025.475510:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757025.475513:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757025.475515:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475517:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757025.475518:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757025.475519:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757025.475521:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757025.475523:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757025.475524:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757025.475527:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757025.475528:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757025.475532:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757025.475534:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475536:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757025.475541:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757025.475543:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757025.475546:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757025.475548:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757027.265419:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757027.265425:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757027.265430:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757028.289520:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757028.289529:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757028.289536:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757029.313583:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757029.313592:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757029.313600:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757030.593350:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757030.593366:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757030.593370:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757030.593375:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757030.593383:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757030.593395:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10085 00000100:00080000:2.0:1787757030.593412:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757030.593418:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757030.593421:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757030.593427:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757030.593431:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757030.593435:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757030.593438:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757030.593441:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757030.593443:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757030.593451:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757030.593455:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757030.593459:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757030.593464:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757030.593469:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.593472:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757030.593475:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757030.593496:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757030.593498:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.593503:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757030.593507:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757030.593515:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.593534:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757030.593548:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659607168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757030.593564:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757030.593570:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.593599:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757030.593617:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757030.593625:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757030.593628:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757030.593631:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757030.593638:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757030.593645:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10085 00000100:00080000:2.0:1787757030.593676:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757030.593684:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757030.593686:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757030.593688:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757030.593697:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757030.593700:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757030.593705:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757030.593711:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757030.593712:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757030.593717:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757030.593722:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757030.593725:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757030.593732:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:2.0:1787757030.593740:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757030.593742:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.593752:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757030.593759:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757030.593759:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757030.593771:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757030.593775:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757030.593779:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757030.593783:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.593789:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:3.0:1787757030.593795:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659607168:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1787757030.593799:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757030.593810:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.593811:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757030.593813:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.593820:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757030.593824:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757030.593826:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757030.593829:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.593834:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757030.593836:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757030.593841:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757030.593844:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.593846:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.593855:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757030.593869:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659607296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757030.593877:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757030.593881:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757030.593883:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:2.0:1787757030.593890:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.593895:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757030.593898:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757030.593905:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757030.593910:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757030.593913:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757030.593915:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757030.593919:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757030.593923:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10085 00000100:00080000:2.0:1787757030.593961:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757030.593964:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757030.593966:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757030.593969:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757030.593972:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757030.593978:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000800:00000100:3.0:1787757030.593980:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787757030.593986:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757030.593992:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757030.593994:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757030.594001:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757030.594005:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594007:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787757030.594008:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757030.594017:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757030.594019:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787757030.594022:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594026:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.594029:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757030.594035:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:2.0:1787757030.594037:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757030.594044:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.594047:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594048:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757030.594055:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757030.594058:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304758465 02000000:00000001:2.0:1787757030.594060:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.594069:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757030.594070:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757030.594073:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.594080:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594082:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:2.0:1787757030.594090:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659607424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757030.594096:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304758465 00000100:00000001:3.0:1787757030.594111:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757030.594113:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757030.594119:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594120:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757030.594123:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594127:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757030.594130:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787757030.594135:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:2.0:1787757030.594137:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757030.594142:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757030.594144:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757030.594152:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757030/real 1787757030] req@ffff8ffb47a38380 x1874588659607168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757030.594153:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757030.594165:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10085 00000100:00080000:2.0:1787757030.594175:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000400:00000001:3.0:1787757030.594226:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757030.594229:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:3.0:1787757030.594232:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757030.594235:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757030.594241:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787757030.594243:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000020:00000001:2.0:1787757030.594244:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757030.594251:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757030.594253:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757030.594258:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757030.594260:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757030.594266:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757030.594268:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757030.594274:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757030.594275:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757030.594289:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1787757030.594293:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659607168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757030.594295:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757030.594303:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757030.594310:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757030.594316:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.594319:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757030.594321:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757030.594324:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757030.594328:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594333:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757030.594337:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757030.594343:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594343:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757030.594351:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:3.0:1787757030.594354:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757030.594356:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787757030.594359:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:2.0:1787757030.594363:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594365:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594371:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757030.594373:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594375:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757030.594385:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757030.594392:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659607552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757030.594399:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757030.594408:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757030.594419:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:3.0:1787757030.594421:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659607168:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757030.594429:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594435:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757030.594436:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594445:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757030.594451:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757030.594455:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757030.594461:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757030.594464:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757030.594467:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757030.594475:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757030.594478:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757030.594483:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757030.594486:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594489:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594500:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757030.594502:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594505:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757030.594508:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757030.594510:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757030.594512:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757030.594514:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.594517:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757030.594520:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757030.594523:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757030.594528:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607296:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757030.594532:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757030.594535:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757030.594537:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.594540:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757030.594544:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594552:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757030.594554:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:3.0:1787757030.594560:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757030.594579:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757030.594585:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757030.594588:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757030.594590:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594592:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757030.594596:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594602:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304757774 00000100:00000001:3.0:1787757030.594606:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304758465 00000100:00000001:3.0:1787757030.594609:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304757774 00000100:00000001:3.0:1787757030.594612:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304758465 00000100:00000001:3.0:1787757030.594615:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757030.594617:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594620:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757030.594625:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757030/real 1787757030] req@ffff8ffb47a38000 x1874588659607296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757030.594638:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757030.594640:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757030.594643:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757030.594645:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757030.594647:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594649:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757030.594678:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757030.594687:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659607296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757030.594702:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594707:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757030.594712:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594715:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757030.594718:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594722:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757030.594750:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757030.594756:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757030.594763:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757030.594768:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757030.594773:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607296:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757030.594779:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757030.594781:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757030.594783:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.594786:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757030.594789:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757030.594792:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757030.594795:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607424:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757030.594799:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757030.594802:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757030.594805:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.594808:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757030.594812:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594820:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757030.594822:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:3.0:1787757030.594828:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757030.594845:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757030.594851:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757030.594854:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757030.594856:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594858:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757030.594861:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594866:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304757771 00000100:00000001:3.0:1787757030.594869:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304758465 00000100:00000001:3.0:1787757030.594872:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304757771 00000100:00000001:3.0:1787757030.594876:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304758465 00000100:00000001:3.0:1787757030.594878:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757030.594881:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.594883:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757030.594888:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757030/real 1787757030] req@ffff8ffb47a3b480 x1874588659607424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757030.594898:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757030.594899:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757030.594902:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757030.594904:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757030.594906:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594908:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757030.594910:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757030.594915:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659607424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757030.594923:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594926:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757030.594930:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594932:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757030.594934:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.594936:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757030.594955:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757030.594958:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757030.594963:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757030.594965:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757030.594969:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607424:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757030.594973:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757030.594975:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757030.594977:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.594979:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757030.594981:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757030.594983:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757030.594986:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757030.594990:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757030.594992:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757030.594995:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757030.594997:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757030.595001:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.595006:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757030.595008:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:3.0:1787757030.595013:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757030.595024:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757030.595029:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757030.595031:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757030.595034:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.595036:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757030.595039:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.595043:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304757771 00000100:00000001:3.0:1787757030.595046:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304758465 00000100:00000001:3.0:1787757030.595050:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304757771 00000100:00000001:3.0:1787757030.595053:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304758465 00000100:00000001:3.0:1787757030.595056:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757030.595058:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.595060:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757030.595065:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757030/real 1787757030] req@ffff8ffb47a39180 x1874588659607552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757030.595074:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757030.595075:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757030.595078:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757030.595079:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757030.595081:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595083:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757030.595084:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757030.595089:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659607552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757030.595098:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595100:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757030.595103:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595106:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757030.595107:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595109:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757030.595128:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757030.595130:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757030.595134:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757030.595137:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757030.595141:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757030.595146:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595148:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757030.595151:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757030.595153:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757030.595156:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757030.595158:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757030.595160:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757030.595162:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757030.595164:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757030.595167:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757030.595169:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595171:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757030.595173:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757030.595175:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757030.595177:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757030.595178:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757030.595180:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757030.595212:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757030.595214:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757030.595218:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757030.595220:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595222:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757030.595223:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757030.595225:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757030.595227:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757030.595229:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757030.595230:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757030.595232:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757030.595234:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757030.595237:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757030.595239:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595242:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757030.595246:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757030.595248:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757030.595250:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757030.595252:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757031.361511:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757031.361521:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757031.361528:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757032.385591:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757032.385601:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757032.385608:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757033.409474:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757033.409484:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757033.409493:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757035.457460:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757035.457470:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757035.457478:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787757035.713847:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757035.713858:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757035.713865:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757035.713871:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757035.713876:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757035.713889:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757035.713896:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757035.713905:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757035.713919:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659607680:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757035.713935:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757035.713944:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757035.713949:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757035.713958:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757035.713970:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714000:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757035.714008:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:3.0:1787757035.714022:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757035.714111:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757035.714142:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757035.714153:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757035.714160:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714168:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757035.714228:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714237:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304763586 00000100:00000001:3.0:1787757035.714255:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757035.714271:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304763586 00000100:00000001:3.0:1787757035.714284:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757035.714292:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757035.714298:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714310:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757035.714326:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757035/real 1787757035] req@ffff8ffb4790c380 x1874588659607680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757035.714357:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757035.714367:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757035.714379:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757035.714385:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757035.714391:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.714403:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757035.714408:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757035.714420:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659607680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757035.714452:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.714460:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757035.714469:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.714478:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757035.714483:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.714490:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757035.714498:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757035.714504:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757035.714515:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757035.714520:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757035.714535:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659607680:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757035.714549:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.714583:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757035.714588:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757035.714595:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757035.714604:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757035.714612:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757035.714621:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757035.714635:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757035.714644:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757035.714651:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757035.714661:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.714672:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.714688:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757035.714694:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714701:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757035.714705:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757035.714710:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757035.714717:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757035.714720:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757035.714724:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757035.714730:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757035.714737:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757035.714747:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607808:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757035.714758:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757035.714766:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757035.714773:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757035.714777:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757035.714784:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714797:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757035.714803:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:3.0:1787757035.714818:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757035.714847:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757035.714856:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757035.714863:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757035.714869:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714874:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757035.714883:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714893:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304762894 00000100:00000001:3.0:1787757035.714900:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304763586 00000100:00000001:3.0:1787757035.714908:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304762894 00000100:00000001:3.0:1787757035.714918:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304763586 00000100:00000001:3.0:1787757035.714921:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757035.714926:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.714935:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757035.714943:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757035/real 1787757035] req@ffff8ffb4790f100 x1874588659607808/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757035.714983:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757035.714987:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757035.714995:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757035.715000:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757035.715004:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715011:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757035.715018:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757035.715037:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659607808/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757035.715069:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715079:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757035.715085:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715089:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757035.715093:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715097:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757035.715126:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757035.715131:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757035.715137:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757035.715140:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757035.715145:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607808:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757035.715151:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757035.715154:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757035.715156:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757035.715159:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757035.715162:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757035.715164:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757035.715168:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757035.715172:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757035.715175:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757035.715178:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757035.715210:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757035.715216:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715225:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757035.715226:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:3.0:1787757035.715233:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757035.715251:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757035.715257:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757035.715261:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757035.715263:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715265:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757035.715269:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715273:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304762892 00000100:00000001:3.0:1787757035.715276:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304763586 00000100:00000001:3.0:1787757035.715279:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304762892 00000100:00000001:3.0:1787757035.715282:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304763586 00000100:00000001:3.0:1787757035.715285:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757035.715288:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715290:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757035.715296:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757035/real 1787757035] req@ffff8ffb4790d880 x1874588659607936/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757035.715305:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757035.715307:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757035.715309:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757035.715311:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757035.715313:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715316:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757035.715318:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757035.715323:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659607936/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757035.715331:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715334:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757035.715338:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715340:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757035.715342:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715344:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757035.715362:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757035.715366:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757035.715370:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757035.715373:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757035.715378:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659607936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757035.715382:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715384:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757035.715386:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757035.715388:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757035.715390:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757035.715392:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757035.715394:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757035.715398:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757035.715399:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757035.715402:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757035.715404:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715406:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757035.715408:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757035.715409:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757035.715412:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757035.715413:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757035.715414:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757035.715417:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757035.715418:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757035.715420:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757035.715422:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715424:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715428:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757035.715430:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715432:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757035.715434:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757035.715436:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757035.715437:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757035.715440:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757035.715443:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757035.715445:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757035.715447:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757035.715451:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757035.715455:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757035.715457:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757035.715459:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757035.715461:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757035.715465:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715470:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757035.715472:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:3.0:1787757035.715477:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757035.715490:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757035.715496:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757035.715498:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757035.715500:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715502:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757035.715506:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715509:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304762892 00000100:00000001:3.0:1787757035.715512:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304763586 00000100:00000001:3.0:1787757035.715515:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304762892 00000100:00000001:3.0:1787757035.715518:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304763586 00000100:00000001:3.0:1787757035.715521:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757035.715524:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715526:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757035.715530:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757035/real 1787757035] req@ffff8ffb4790e680 x1874588659608064/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757035.715541:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757035.715543:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757035.715547:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757035.715550:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757035.715553:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715557:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757035.715582:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757035.715592:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659608064/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757035.715607:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715611:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757035.715617:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715620:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757035.715623:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715627:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757035.715654:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757035.715659:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757035.715665:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757035.715669:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757035.715673:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757035.715679:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715681:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757035.715683:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757035.715685:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757035.715689:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757035.715691:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757035.715693:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757035.715695:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757035.715697:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757035.715700:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757035.715702:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715704:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757035.715708:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757035.715711:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757035.715713:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757035.715716:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757036.481500:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757036.481506:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757036.481511:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757037.505455:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757037.505461:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757037.505467:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757039.553657:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757039.553670:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757039.553678:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757040.577529:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757040.577538:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757040.577547:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787757040.833643:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757040.833659:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757040.833666:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757040.833681:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757040.833691:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757040.833707:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757040.833715:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757040.833723:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757040.833746:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659608192:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757040.833765:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757040.833773:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757040.833778:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757040.833782:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757040.833795:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.833829:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757040.833837:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:3.0:1787757040.833846:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757040.833926:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757040.833951:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757040.833965:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757040.833977:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.833983:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757040.834009:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834023:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304768705 00000100:00000001:3.0:1787757040.834042:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757040.834060:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304768705 00000100:00000001:3.0:1787757040.834071:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757040.834085:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757040.834096:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834106:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757040.834120:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757040/real 1787757040] req@ffff8ffb4790d180 x1874588659608192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757040.834173:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757040.834178:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757040.834244:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757040.834259:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757040.834270:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834277:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757040.834284:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757040.834298:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659608192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757040.834348:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834363:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757040.834375:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834383:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757040.834388:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834395:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757040.834408:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757040.834425:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757040.834453:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757040.834498:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757040.834514:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659608192:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757040.834534:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834547:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757040.834556:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757040.834564:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757040.834575:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757040.834587:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757040.834591:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757040.834601:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757040.834604:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757040.834610:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757040.834613:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834617:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834629:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757040.834632:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834636:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757040.834638:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757040.834641:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757040.834644:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757040.834646:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757040.834650:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757040.834653:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757040.834657:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757040.834662:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608320:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757040.834668:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757040.834671:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757040.834674:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757040.834676:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757040.834681:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834690:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757040.834692:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:3.0:1787757040.834699:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757040.834720:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757040.834727:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757040.834731:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757040.834733:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834735:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757040.834739:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834744:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304768014 00000100:00000001:3.0:1787757040.834748:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304768705 00000100:00000001:3.0:1787757040.834752:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304768014 00000100:00000001:3.0:1787757040.834755:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304768705 00000100:00000001:3.0:1787757040.834758:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757040.834762:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834764:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757040.834770:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757040/real 1787757040] req@ffff8ffb4790fb80 x1874588659608320/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757040.834783:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757040.834785:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757040.834789:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757040.834791:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757040.834792:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834795:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757040.834797:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757040.834802:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659608320/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757040.834811:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834813:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757040.834816:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834818:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757040.834820:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834822:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757040.834845:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757040.834848:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757040.834852:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757040.834855:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757040.834859:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608320:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757040.834864:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757040.834866:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757040.834868:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757040.834872:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757040.834874:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757040.834876:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757040.834880:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608448:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757040.834885:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757040.834887:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757040.834889:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757040.834892:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757040.834895:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834902:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757040.834904:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:3.0:1787757040.834909:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757040.834921:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757040.834926:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757040.834929:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757040.834931:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834933:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757040.834936:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834941:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304768011 00000100:00000001:3.0:1787757040.834944:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304768705 00000100:00000001:3.0:1787757040.834947:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304768011 00000100:00000001:3.0:1787757040.834950:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304768705 00000100:00000001:3.0:1787757040.834953:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757040.834955:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.834957:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757040.834962:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757040/real 1787757040] req@ffff8ffb4790ed80 x1874588659608448/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757040.834971:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757040.834972:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757040.834974:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757040.834976:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757040.834978:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.834980:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757040.834981:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757040.834985:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659608448/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757040.835002:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835005:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757040.835008:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835010:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757040.835012:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835014:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757040.835033:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757040.835036:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757040.835039:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757040.835042:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757040.835045:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608448:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757040.835050:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757040.835053:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757040.835055:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757040.835058:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757040.835060:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757040.835063:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757040.835066:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757040.835070:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757040.835072:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757040.835073:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757040.835075:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757040.835079:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.835084:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757040.835085:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:3.0:1787757040.835090:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757040.835101:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757040.835106:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757040.835109:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757040.835111:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.835113:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757040.835116:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.835121:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304768011 00000100:00000001:3.0:1787757040.835124:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304768705 00000100:00000001:3.0:1787757040.835127:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304768011 00000100:00000001:3.0:1787757040.835130:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304768705 00000100:00000001:3.0:1787757040.835133:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757040.835135:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.835137:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757040.835142:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757040/real 1787757040] req@ffff8ffb4790ca80 x1874588659608576/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757040.835151:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757040.835152:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757040.835154:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757040.835156:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757040.835158:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835160:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757040.835161:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757040.835165:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659608576/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757040.835174:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835177:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757040.835180:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835182:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757040.835184:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835220:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757040.835239:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757040.835243:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757040.835247:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757040.835250:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757040.835255:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757040.835260:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835263:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757040.835264:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757040.835267:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757040.835270:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757040.835271:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757040.835273:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757040.835277:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757040.835279:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757040.835282:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757040.835284:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835286:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757040.835288:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757040.835289:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757040.835292:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757040.835294:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757040.835295:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757040.835297:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757040.835298:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757040.835301:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757040.835302:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835304:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757040.835305:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757040.835307:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757040.835308:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757040.835310:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757040.835311:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757040.835313:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757040.835314:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757040.835316:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757040.835318:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835320:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757040.835324:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757040.835325:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757040.835328:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757040.835330:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757041.601451:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757041.601503:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757041.601509:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757043.649457:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757043.649466:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757043.649473:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757044.673467:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757044.673475:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757044.673483:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757045.697452:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757045.697459:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757045.697465:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787757045.953699:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757045.953713:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757045.953724:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757045.953733:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757045.953743:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757045.953757:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757045.953770:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757045.953786:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757045.953807:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659608704:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757045.953822:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757045.953833:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757045.953843:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757045.953856:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757045.953866:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.953901:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757045.953910:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:3.0:1787757045.953921:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757045.953993:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757045.954021:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757045.954032:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757045.954038:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954042:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757045.954053:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954062:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304773825 00000100:00000001:3.0:1787757045.954084:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757045.954101:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304773825 00000100:00000001:3.0:1787757045.954116:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757045.954130:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757045.954140:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954148:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757045.954164:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757045/real 1787757045] req@ffff8ffb4a537b80 x1874588659608704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757045.954253:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757045.954260:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757045.954270:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757045.954279:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757045.954285:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954289:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757045.954293:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757045.954303:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659608704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757045.954315:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954321:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757045.954325:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954327:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757045.954329:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954332:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757045.954335:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757045.954339:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757045.954344:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757045.954348:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757045.954353:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659608704:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757045.954361:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954364:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757045.954366:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757045.954370:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757045.954375:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757045.954378:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757045.954413:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757045.954422:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757045.954425:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757045.954429:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757045.954431:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954434:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954445:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757045.954448:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954451:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757045.954453:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757045.954456:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757045.954458:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757045.954460:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757045.954463:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757045.954466:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757045.954469:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757045.954474:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608832:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757045.954478:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757045.954482:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757045.954484:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757045.954486:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757045.954490:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954499:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757045.954501:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:3.0:1787757045.954510:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757045.954537:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757045.954547:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757045.954551:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757045.954554:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954557:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757045.954563:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954572:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304773134 00000100:00000001:3.0:1787757045.954578:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304773825 00000100:00000001:3.0:1787757045.954583:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304773134 00000100:00000001:3.0:1787757045.954587:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304773825 00000100:00000001:3.0:1787757045.954592:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757045.954597:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954601:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757045.954611:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757045/real 1787757045] req@ffff8ffb4a536300 x1874588659608832/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757045.954631:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757045.954634:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757045.954641:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757045.954644:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757045.954648:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954652:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757045.954656:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757045.954665:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659608832/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757045.954681:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954686:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757045.954690:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954694:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757045.954697:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954701:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757045.954730:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757045.954737:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757045.954746:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757045.954751:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757045.954757:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608832:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757045.954766:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757045.954768:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757045.954772:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757045.954776:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757045.954781:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757045.954785:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757045.954792:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608960:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757045.954799:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757045.954804:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757045.954807:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757045.954811:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757045.954817:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954829:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757045.954832:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757045.954840:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757045.954860:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757045.954867:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757045.954871:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757045.954873:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954875:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757045.954879:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954884:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304773131 00000100:00000001:3.0:1787757045.954887:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304773825 00000100:00000001:3.0:1787757045.954890:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304773131 00000100:00000001:3.0:1787757045.954894:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304773825 00000100:00000001:3.0:1787757045.954896:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757045.954899:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.954901:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757045.954907:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757045/real 1787757045] req@ffff8ffb4a537800 x1874588659608960/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757045.954916:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757045.954918:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757045.954921:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757045.954923:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757045.954925:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954928:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757045.954929:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757045.954934:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659608960/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757045.954952:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954955:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757045.954958:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954960:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757045.954961:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.954964:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757045.954983:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757045.954986:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757045.954989:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757045.954991:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757045.954995:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659608960:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757045.955000:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757045.955002:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757045.955004:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757045.955007:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757045.955009:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757045.955012:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757045.955016:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659609088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757045.955020:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757045.955022:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757045.955024:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757045.955027:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757045.955030:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.955035:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757045.955037:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757045.955041:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757045.955053:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757045.955058:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757045.955061:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757045.955062:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.955065:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757045.955068:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.955072:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304773131 00000100:00000001:3.0:1787757045.955075:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304773825 00000100:00000001:3.0:1787757045.955078:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304773131 00000100:00000001:3.0:1787757045.955081:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304773825 00000100:00000001:3.0:1787757045.955084:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757045.955086:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.955088:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757045.955093:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757045/real 1787757045] req@ffff8ffb4a534700 x1874588659609088/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757045.955102:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757045.955103:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757045.955105:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757045.955107:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757045.955109:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955111:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757045.955112:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757045.955117:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659609088/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757045.955125:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955128:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757045.955131:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955133:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757045.955135:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955137:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757045.955155:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757045.955158:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757045.955162:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757045.955164:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757045.955169:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659609088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757045.955175:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955178:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757045.955181:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757045.955223:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757045.955229:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757045.955232:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757045.955235:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757045.955242:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757045.955245:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757045.955249:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757045.955252:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955255:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757045.955258:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757045.955261:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757045.955264:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757045.955267:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757045.955270:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757045.955273:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757045.955276:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757045.955279:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757045.955281:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955285:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757045.955287:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757045.955290:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757045.955293:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757045.955296:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757045.955298:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757045.955302:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757045.955304:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757045.955308:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757045.955310:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955314:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757045.955321:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757045.955325:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757045.955330:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757045.955334:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757047.745562:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757047.745568:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757047.745574:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757048.769363:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757048.769367:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757048.769370:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757049.793538:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757049.793545:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757049.793550:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787757051.073807:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757051.073816:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757051.073825:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757051.073835:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757051.073844:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757051.073861:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757051.073871:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757051.073882:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757051.073899:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659609216:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757051.073920:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757051.073930:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757051.073938:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757051.073947:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757051.073963:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074002:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757051.074007:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:3.0:1787757051.074019:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757051.074091:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757051.074134:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757051.074147:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757051.074155:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074163:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757051.074175:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074234:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304778946 00000100:00000001:3.0:1787757051.074257:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757051.074277:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304778946 00000100:00000001:3.0:1787757051.074318:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757051.074332:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757051.074346:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074361:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757051.074377:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757051/real 1787757051] req@ffff8ffb4790f480 x1874588659609216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757051.074418:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757051.074424:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757051.074437:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757051.074448:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757051.074453:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074458:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757051.074465:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757051.074477:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659609216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757051.074518:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074527:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757051.074535:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074538:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757051.074540:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074544:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757051.074547:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757051.074551:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757051.074558:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757051.074562:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757051.074567:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659609216:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757051.074574:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074577:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757051.074579:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757051.074583:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757051.074588:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757051.074592:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757051.074595:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757051.074602:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757051.074605:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757051.074608:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757051.074611:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074614:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074625:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757051.074628:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074631:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757051.074633:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757051.074636:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757051.074638:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757051.074640:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757051.074643:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757051.074646:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757051.074649:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757051.074653:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659609344:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757051.074657:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757051.074660:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757051.074662:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757051.074665:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757051.074669:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074677:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757051.074679:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133080064 : -123165576471552 : ffff8ffb4790dc00) 00000100:00000001:3.0:1787757051.074686:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757051.074705:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757051.074711:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757051.074714:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757051.074716:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074718:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757051.074722:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074728:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304778254 00000100:00000001:3.0:1787757051.074732:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304778946 00000100:00000001:3.0:1787757051.074735:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304778254 00000100:00000001:3.0:1787757051.074738:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304778946 00000100:00000001:3.0:1787757051.074741:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757051.074743:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074745:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757051.074751:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757051/real 1787757051] req@ffff8ffb4790dc00 x1874588659609344/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757051.074763:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757051.074765:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757051.074768:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757051.074770:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757051.074772:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074774:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757051.074776:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757051.074781:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790dc00 x1874588659609344/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757051.074790:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074793:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757051.074795:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074797:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757051.074799:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074801:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757051.074824:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757051.074827:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757051.074830:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757051.074833:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757051.074836:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659609344:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757051.074841:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757051.074842:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757051.074844:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757051.074846:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757051.074849:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757051.074851:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757051.074854:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659609472:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757051.074859:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757051.074861:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757051.074863:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757051.074866:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757051.074870:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074875:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757051.074877:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133072896 : -123165576478720 : ffff8ffb4790c000) 00000100:00000001:3.0:1787757051.074883:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757051.074894:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757051.074899:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757051.074901:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757051.074903:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074905:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757051.074908:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074912:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304778251 00000100:00000001:3.0:1787757051.074916:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304778946 00000100:00000001:3.0:1787757051.074919:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304778251 00000100:00000001:3.0:1787757051.074922:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304778946 00000100:00000001:3.0:1787757051.074925:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757051.074927:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.074929:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757051.074934:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757051/real 1787757051] req@ffff8ffb4790c000 x1874588659609472/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757051.074943:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757051.074944:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757051.074947:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757051.074948:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757051.074950:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074952:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757051.074954:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757051.074958:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c000 x1874588659609472/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757051.074966:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074969:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757051.074973:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074975:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757051.074977:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.074979:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757051.074998:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757051.075002:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757051.075006:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757051.075009:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757051.075013:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659609472:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757051.075017:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757051.075018:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757051.075020:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757051.075022:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757051.075024:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757051.075027:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757051.075030:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659609600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757051.075033:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757051.075035:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757051.075037:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757051.075039:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757051.075043:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.075048:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757051.075050:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:3.0:1787757051.075054:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757051.075065:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757051.075070:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757051.075072:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757051.075074:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.075076:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757051.075079:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.075083:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304778251 00000100:00000001:3.0:1787757051.075086:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304778946 00000100:00000001:3.0:1787757051.075090:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304778251 00000100:00000001:3.0:1787757051.075093:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304778946 00000100:00000001:3.0:1787757051.075096:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757051.075098:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.075101:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757051.075105:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757051/real 1787757051] req@ffff8ffb4790ca80 x1874588659609600/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757051.075114:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757051.075115:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757051.075118:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757051.075119:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757051.075121:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075123:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757051.075124:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757051.075129:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659609600/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757051.075136:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075139:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757051.075142:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075144:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757051.075146:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075148:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757051.075168:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757051.075171:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757051.075175:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757051.075178:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757051.075182:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659609600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757051.075219:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075223:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757051.075226:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757051.075229:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757051.075234:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757051.075236:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757051.075239:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757051.075243:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757051.075246:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757051.075250:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757051.075252:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075256:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757051.075258:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757051.075260:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757051.075265:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757051.075267:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757051.075269:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757051.075273:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757051.075275:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757051.075278:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757051.075281:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075285:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757051.075287:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757051.075312:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757051.075318:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757051.075320:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757051.075323:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757051.075326:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757051.075328:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757051.075332:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757051.075334:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075338:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757051.075344:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757051.075346:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757051.075349:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757051.075351:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757051.841447:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757051.841456:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757051.841463:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757053.889434:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757053.889440:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757053.889443:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757055.937509:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757055.937518:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757055.937525:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757056.193356:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757056.193367:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757056.193369:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757056.193372:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757056.193377:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757056.193384:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10110 00000100:00080000:2.0:1787757056.193393:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757056.193397:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757056.193399:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757056.193403:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757056.193405:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757056.193408:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757056.193410:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757056.193412:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757056.193413:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757056.193416:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757056.193418:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757056.193422:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757056.193426:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757056.193430:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193433:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757056.193436:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757056.193449:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757056.193451:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193455:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757056.193457:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757056.193489:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193503:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757056.193515:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659609728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757056.193528:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757056.193533:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193556:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757056.193561:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757056.193566:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757056.193568:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757056.193573:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757056.193578:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757056.193583:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10110 00000100:00080000:2.0:1787757056.193593:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757056.193619:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757056.193626:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757056.193640:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757056.193647:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757056.193656:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757056.193666:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757056.193679:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757056.193686:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757056.193704:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757056.193708:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757056.193711:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757056.193725:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757056.193734:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193738:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757056.193744:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757056.193749:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193752:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757056.193759:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757056.193760:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193762:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757056.193764:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757056.193769:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193777:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757056.193806:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659609856/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757056.193824:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757056.193829:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193837:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757056.193843:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757056.193850:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757056.193874:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757056.193876:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757056.193881:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757056.193885:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10110 00000100:00080000:2.0:1787757056.193890:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757056.193895:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757056.193900:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757056.193903:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757056.193904:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757056.193906:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757056.193907:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757056.193908:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757056.193909:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757056.193912:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757056.193920:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757056.193932:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757056.193945:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757056.193952:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193961:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757056.193966:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757056.193976:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.193987:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757056.194000:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757056.194005:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.194009:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757056.194013:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757056.194019:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.194034:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757056.194049:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659609984/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757056.194106:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757056.194112:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.194125:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757056.194135:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757056.194150:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757056.194153:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757056.194159:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757056.194173:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757056.194181:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10110 00000100:00080000:2.0:1787757056.194187:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757056.194196:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757056.194249:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757056.194252:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757056.194256:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757056.194260:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757056.194267:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757056.194275:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757056.194279:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757056.194288:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757056.194297:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757056.194306:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757056.194314:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757056.194322:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.194326:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757056.194329:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757056.194341:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.194346:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757056.194354:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757056.194355:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.194362:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757056.194372:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757056.194376:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.194388:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757056.194402:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659610112/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757056.194431:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757056.194440:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757056.194456:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757057.985442:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757057.985449:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757057.985454:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757060.033453:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757060.033463:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757060.033470:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757061.313392:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757061.313406:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757061.313411:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757061.313416:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757061.313423:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757061.313433:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10116 00000100:00080000:2.0:1787757061.313460:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757061.313464:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757061.313468:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757061.313474:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757061.313477:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757061.313481:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757061.313485:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757061.313488:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757061.313491:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757061.313497:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757061.313501:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757061.313505:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757061.313510:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757061.313514:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313516:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757061.313519:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757061.313533:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757061.313537:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313541:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757061.313544:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757061.313550:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313565:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757061.313577:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659610240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757061.313591:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757061.313596:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313616:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757061.313620:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757061.313624:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757061.313626:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757061.313629:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757061.313632:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757061.313635:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10116 00000100:00080000:2.0:1787757061.313640:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757061.313643:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757061.313645:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757061.313648:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757061.313651:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757061.313656:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757061.313659:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757061.313662:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757061.313664:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757061.313669:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757061.313672:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757061.313674:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757061.313682:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757061.313690:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313703:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757061.313722:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757061.313747:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313754:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757061.313762:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757061.313768:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313775:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757061.313782:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757061.313791:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313818:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757061.313843:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659610368/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757061.313882:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757061.313890:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.313902:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757061.313915:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757061.313928:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757061.313932:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757061.313935:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757061.313940:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757061.313946:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10116 00000100:00080000:2.0:1787757061.313953:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757061.313956:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757061.313958:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757061.313961:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757061.313964:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757061.313968:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757061.313972:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757061.313975:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757061.313977:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757061.313981:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757061.313986:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757061.313989:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757061.313996:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757061.313999:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314001:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757061.314006:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757061.314010:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314030:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757061.314043:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757061.314048:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314053:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757061.314057:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757061.314065:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314075:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757061.314099:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659610496/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757061.314161:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757061.314172:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314261:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757061.314275:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757061.314294:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757061.314305:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757061.314314:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757061.314331:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757061.314343:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10116 00000100:00080000:2.0:1787757061.314366:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757061.314379:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757061.314393:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757061.314408:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757061.314417:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757061.314423:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757061.314427:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757061.314443:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757061.314446:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757061.314453:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757061.314457:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757061.314460:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757061.314466:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757061.314477:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314481:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757061.314495:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757061.314516:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314528:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757061.314554:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757061.314563:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314579:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757061.314590:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757061.314604:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314632:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757061.314659:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659610624/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757061.314687:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757061.314696:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757061.314704:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757064.129405:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757064.129411:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757064.129416:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787757064.385561:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787757064.385664:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787757064.385693:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757064.385723:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757064.385756:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757064.385762:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757064.385767:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757064.385786:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757064.385796:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757064.385801:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757064.385804:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757064.385808:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757064.385817:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757064.385833:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757064.385838:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757064.385840:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757064.385843:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757064.385851:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757064.385859:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757064.385865:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757064.385868:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757064.385872:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757066.433821:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757066.433835:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757066.433846:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757066.433858:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757066.433869:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757066.433886:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757066.433898:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757066.433914:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757066.433939:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659610752:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757066.433960:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757066.433974:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757066.433983:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757066.433995:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757066.434011:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.434084:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757066.434092:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:3.0:1787757066.434106:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757066.434239:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757066.434266:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757066.434274:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757066.434280:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.434290:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757066.434310:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.434325:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304794306 00000100:00000001:3.0:1787757066.434351:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757066.434370:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304794306 00000100:00000001:3.0:1787757066.434389:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757066.434404:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757066.434416:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.434442:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757066.434466:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757066/real 1787757066] req@ffff8ffb4a535c00 x1874588659610752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757066.434503:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757066.434512:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757066.434527:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757066.434539:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757066.434707:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.434719:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757066.434729:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757066.434756:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659610752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757066.434786:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.434796:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757066.434803:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.434807:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757066.434811:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.434817:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757066.434823:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757066.434829:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757066.434839:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757066.434844:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757066.434853:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659610752:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757066.434862:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.434867:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757066.434871:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757066.434877:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757066.434885:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757066.434889:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757066.434893:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757066.434901:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757066.434905:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757066.434911:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757066.434914:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.434919:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.434935:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757066.434940:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.434945:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757066.434948:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757066.434951:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757066.434955:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757066.434958:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757066.434962:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757066.434966:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757066.434972:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757066.434979:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659610880:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757066.434988:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757066.434992:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757066.434996:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757066.435000:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757066.435006:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435048:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757066.435052:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:3.0:1787757066.435064:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757066.435094:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757066.435106:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757066.435112:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757066.435115:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435118:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757066.435124:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435133:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304793614 00000100:00000001:3.0:1787757066.435139:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304794306 00000100:00000001:3.0:1787757066.435144:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304793614 00000100:00000001:3.0:1787757066.435148:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304794306 00000100:00000001:3.0:1787757066.435152:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757066.435157:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435162:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757066.435172:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757066/real 1787757066] req@ffff8ffb4a534a80 x1874588659610880/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757066.435233:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757066.435236:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757066.435243:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757066.435247:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757066.435249:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435254:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757066.435257:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757066.435267:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659610880/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757066.435283:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435288:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757066.435293:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435296:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757066.435299:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435303:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757066.435333:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757066.435340:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757066.435348:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757066.435352:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757066.435359:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659610880:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757066.435368:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757066.435370:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757066.435373:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757066.435378:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757066.435382:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757066.435387:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757066.435393:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611008:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757066.435400:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757066.435405:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757066.435408:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757066.435412:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757066.435419:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435429:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757066.435432:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757066.435440:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757066.435464:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757066.435472:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757066.435490:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757066.435494:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435499:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757066.435505:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435512:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304793612 00000100:00000001:3.0:1787757066.435517:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304794306 00000100:00000001:3.0:1787757066.435522:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304793612 00000100:00000001:3.0:1787757066.435526:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304794306 00000100:00000001:3.0:1787757066.435530:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757066.435535:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435538:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757066.435548:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757066/real 1787757066] req@ffff8ffb4a534700 x1874588659611008/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757066.435584:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757066.435586:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757066.435593:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757066.435596:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757066.435599:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435602:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757066.435605:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757066.435612:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659611008/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757066.435625:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435629:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757066.435634:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435637:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757066.435639:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435643:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757066.435667:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757066.435673:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757066.435679:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757066.435684:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757066.435692:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611008:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757066.435698:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757066.435701:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757066.435704:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757066.435708:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757066.435711:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757066.435716:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757066.435722:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757066.435727:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757066.435731:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757066.435734:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757066.435737:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757066.435744:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435754:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757066.435756:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757066.435763:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757066.435788:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757066.435797:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757066.435801:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757066.435804:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435807:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757066.435812:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435819:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304793612 00000100:00000001:3.0:1787757066.435824:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304794306 00000100:00000001:3.0:1787757066.435827:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304793612 00000100:00000001:3.0:1787757066.435831:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304794306 00000100:00000001:3.0:1787757066.435853:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757066.435858:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.435862:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757066.435871:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757066/real 1787757066] req@ffff8ffb4a537800 x1874588659611136/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757066.435918:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757066.435920:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757066.435925:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757066.435927:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757066.435929:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435931:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757066.435933:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757066.435938:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659611136/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757066.435947:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435950:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757066.435954:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435956:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757066.435958:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.435962:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757066.435987:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757066.435992:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757066.435999:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757066.436004:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757066.436011:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757066.436042:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.436048:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757066.436050:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757066.436054:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757066.436059:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757066.436062:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757066.436065:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757066.436071:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757066.436073:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757066.436077:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757066.436080:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.436084:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757066.436086:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757066.436088:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757066.436092:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757066.436094:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757066.436096:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757066.436100:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757066.436102:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757066.436104:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757066.436107:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.436110:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757066.436112:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757066.436115:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757066.436118:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757066.436121:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757066.436123:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757066.436127:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757066.436129:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757066.436132:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757066.436135:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.436139:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757066.436146:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757066.436148:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757066.436152:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757066.436155:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757067.201479:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757067.201502:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757067.201510:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757068.225431:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757068.225438:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757068.225444:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757071.297551:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757071.297558:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757071.297563:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787757071.553668:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757071.553683:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757071.553695:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757071.553707:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757071.553718:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757071.553734:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757071.553740:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757071.553749:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757071.553760:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659611264:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757071.553771:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757071.553782:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757071.553789:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757071.553801:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757071.553813:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.553845:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757071.553851:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:3.0:1787757071.553860:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757071.553958:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757071.553995:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757071.554005:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757071.554015:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554023:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757071.554037:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554054:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304799425 00000100:00000001:3.0:1787757071.554085:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757071.554107:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304799425 00000100:00000001:3.0:1787757071.554127:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757071.554144:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757071.554154:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554161:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757071.554176:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757071/real 1787757071] req@ffff8ffb4a536300 x1874588659611264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757071.554294:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757071.554302:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757071.554316:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757071.554326:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757071.554330:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554337:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757071.554343:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757071.554353:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659611264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757071.554375:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554380:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757071.554385:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554387:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757071.554388:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554392:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757071.554394:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757071.554399:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757071.554404:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757071.554407:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757071.554412:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659611264:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757071.554420:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554423:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757071.554426:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757071.554430:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757071.554435:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757071.554438:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757071.554442:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757071.554449:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757071.554452:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757071.554456:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757071.554458:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554461:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554472:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757071.554475:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554478:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757071.554480:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757071.554483:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757071.554485:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757071.554487:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757071.554490:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757071.554493:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757071.554496:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757071.554501:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611392:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757071.554505:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757071.554508:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757071.554511:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757071.554514:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757071.554518:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554527:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757071.554529:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:3.0:1787757071.554536:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757071.554556:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757071.554563:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757071.554566:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757071.554568:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554570:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757071.554574:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554581:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304798734 00000100:00000001:3.0:1787757071.554585:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304799425 00000100:00000001:3.0:1787757071.554588:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304798734 00000100:00000001:3.0:1787757071.554592:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304799425 00000100:00000001:3.0:1787757071.554594:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757071.554597:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554599:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757071.554605:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757071/real 1787757071] req@ffff8ffb4a537b80 x1874588659611392/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757071.554615:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757071.554616:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757071.554620:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757071.554622:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757071.554624:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554626:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757071.554628:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757071.554633:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659611392/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757071.554651:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554654:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757071.554657:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554659:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757071.554661:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554663:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757071.554685:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757071.554688:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757071.554692:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757071.554694:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757071.554698:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611392:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757071.554703:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757071.554705:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757071.554707:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757071.554710:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757071.554713:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757071.554715:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757071.554718:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611520:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757071.554722:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757071.554725:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757071.554727:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757071.554729:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757071.554733:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554739:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757071.554741:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757071.554747:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757071.554759:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757071.554764:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757071.554766:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757071.554768:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554771:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757071.554774:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554778:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304798731 00000100:00000001:3.0:1787757071.554781:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304799425 00000100:00000001:3.0:1787757071.554784:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304798731 00000100:00000001:3.0:1787757071.554787:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304799425 00000100:00000001:3.0:1787757071.554790:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757071.554792:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554794:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757071.554799:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757071/real 1787757071] req@ffff8ffb4a537800 x1874588659611520/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757071.554807:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757071.554809:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757071.554811:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757071.554813:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757071.554815:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554817:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757071.554818:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757071.554823:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659611520/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757071.554831:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554834:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757071.554837:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554839:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757071.554841:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.554843:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757071.554862:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757071.554865:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757071.554869:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757071.554871:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757071.554876:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611520:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757071.554881:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757071.554883:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757071.554885:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757071.554887:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757071.554889:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757071.554891:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757071.554895:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757071.554898:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757071.554900:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757071.554902:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757071.554904:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757071.554908:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554913:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757071.554915:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757071.554920:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757071.554959:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757071.554966:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757071.554969:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757071.554971:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554974:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757071.554977:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554982:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304798731 00000100:00000001:3.0:1787757071.554986:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304799425 00000100:00000001:3.0:1787757071.554989:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304798731 00000100:00000001:3.0:1787757071.554992:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304799425 00000100:00000001:3.0:1787757071.554995:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757071.554997:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.554999:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757071.555005:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757071/real 1787757071] req@ffff8ffb4a534700 x1874588659611648/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757071.555023:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757071.555024:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757071.555027:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757071.555029:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757071.555031:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555034:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757071.555037:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757071.555044:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659611648/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757071.555058:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555063:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757071.555067:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555070:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757071.555072:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555076:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757071.555100:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757071.555106:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757071.555113:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757071.555118:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757071.555125:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659611648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757071.555133:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555137:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757071.555140:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757071.555143:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757071.555148:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757071.555150:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757071.555152:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757071.555155:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757071.555157:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757071.555160:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757071.555162:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555165:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757071.555167:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757071.555169:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757071.555171:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757071.555173:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757071.555174:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757071.555178:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757071.555179:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757071.555181:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757071.555212:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555215:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757071.555216:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757071.555218:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757071.555220:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757071.555221:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757071.555223:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757071.555225:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757071.555227:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757071.555229:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757071.555231:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555234:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757071.555239:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757071.555241:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757071.555243:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757071.555245:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757072.321511:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757072.321520:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757072.321536:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757075.393461:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757075.393468:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757075.393473:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757076.417451:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757076.417460:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757076.417467:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757076.673369:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757076.673381:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757076.673385:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757076.673389:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757076.673395:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757076.673405:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10131 00000100:00080000:3.0:1787757076.673419:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757076.673423:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757076.673426:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757076.673431:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757076.673434:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757076.673438:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757076.673442:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757076.673444:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757076.673447:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757076.673468:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757076.673474:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757076.673479:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757076.673484:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757076.673491:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.673495:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757076.673500:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757076.673515:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757076.673519:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.673524:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757076.673529:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757076.673537:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.673555:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757076.673570:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659611776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757076.673585:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757076.673591:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.673616:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757076.673621:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757076.673625:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757076.673627:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757076.673630:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757076.673635:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757076.673639:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10131 00000100:00080000:3.0:1787757076.673646:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757076.673650:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757076.673653:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757076.673657:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757076.673670:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757076.673674:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757076.673682:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757076.673688:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757076.673693:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757076.673707:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757076.673721:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757076.673741:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757076.673754:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757076.673782:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.673808:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757076.673819:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757076.673864:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.673872:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757076.673880:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757076.673885:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.673903:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757076.673908:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757076.673916:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.673934:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757076.673961:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659611904/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757076.673981:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757076.673993:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674013:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757076.674022:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757076.674031:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757076.674034:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757076.674038:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757076.674043:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757076.674049:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10131 00000100:00080000:3.0:1787757076.674057:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757076.674060:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757076.674063:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757076.674067:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757076.674069:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757076.674077:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757076.674089:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757076.674095:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757076.674099:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757076.674106:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757076.674111:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757076.674118:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757076.674126:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757076.674136:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674141:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757076.674147:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757076.674159:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674162:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757076.674176:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757076.674178:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674235:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757076.674240:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757076.674246:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674266:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757076.674296:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659612032/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757076.674397:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757076.674402:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674410:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757076.674417:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757076.674428:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757076.674434:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757076.674439:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757076.674447:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757076.674458:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10131 00000100:00080000:3.0:1787757076.674481:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757076.674499:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757076.674507:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757076.674524:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757076.674543:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757076.674552:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757076.674566:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757076.674577:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757076.674586:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757076.674604:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757076.674610:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757076.674618:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757076.674632:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757076.674644:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674650:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757076.674657:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757076.674667:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674675:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757076.674690:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757076.674696:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674717:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757076.674729:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757076.674735:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674746:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757076.674780:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659612160/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757076.674824:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757076.674862:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757076.674884:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757079.489463:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757079.489471:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757079.489476:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757080.513437:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757080.513443:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757080.513459:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757081.793319:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757081.793335:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757081.793339:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757081.793344:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757081.793352:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757081.793363:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10136 00000100:00080000:3.0:1787757081.793379:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757081.793384:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757081.793387:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757081.793394:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757081.793398:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757081.793402:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757081.793406:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757081.793409:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757081.793412:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757081.793418:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757081.793421:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757081.793427:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757081.793432:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757081.793437:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793440:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757081.793443:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757081.793460:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757081.793462:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793467:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757081.793472:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757081.793478:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793496:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757081.793509:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659612288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757081.793524:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757081.793530:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793556:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757081.793563:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757081.793570:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757081.793574:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757081.793577:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757081.793583:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757081.793588:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10136 00000100:00080000:3.0:1787757081.793595:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757081.793599:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757081.793603:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757081.793609:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757081.793613:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757081.793617:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757081.793623:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757081.793628:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757081.793631:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757081.793639:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757081.793647:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757081.793657:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757081.793675:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757081.793689:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793694:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757081.793710:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757081.793722:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793727:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757081.793740:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757081.793776:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793786:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757081.793801:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757081.793806:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793833:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757081.793859:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659612416/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757081.793882:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757081.793886:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.793894:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757081.793914:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757081.793932:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757081.793946:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757081.793949:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757081.793954:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757081.793958:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10136 00000100:00080000:3.0:1787757081.793968:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757081.793970:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757081.793972:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757081.793975:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757081.793986:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757081.793989:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757081.793992:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757081.793997:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757081.794001:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757081.794006:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757081.794010:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757081.794012:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757081.794018:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757081.794029:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794034:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757081.794050:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757081.794054:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794057:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757081.794063:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757081.794065:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794068:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757081.794071:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757081.794075:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794087:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757081.794103:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659612544/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757081.794129:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757081.794135:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794146:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757081.794151:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757081.794156:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757081.794158:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757081.794162:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757081.794175:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757081.794228:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10136 00000100:00080000:3.0:1787757081.794240:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757081.794245:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757081.794250:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757081.794257:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757081.794260:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757081.794264:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757081.794270:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757081.794276:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757081.794286:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757081.794309:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757081.794322:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757081.794327:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757081.794338:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757081.794352:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794367:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757081.794375:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757081.794382:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794386:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757081.794399:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757081.794406:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794415:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757081.794420:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757081.794428:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794443:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757081.794467:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659612672/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757081.794505:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757081.794512:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757081.794524:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757083.585438:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757083.585445:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757083.585452:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757084.609476:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757084.609487:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757084.609496:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757086.657412:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757086.657418:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757086.657422:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757086.913264:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757086.913282:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757086.913286:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757086.913292:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757086.913299:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757086.913313:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10141 00000100:00080000:3.0:1787757086.913334:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757086.913341:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757086.913345:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757086.913365:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757086.913370:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757086.913377:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757086.913383:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757086.913387:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757086.913390:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757086.913400:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757086.913407:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757086.913412:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757086.913419:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757086.913424:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913428:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757086.913432:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757086.913447:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757086.913450:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913455:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757086.913459:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757086.913465:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913485:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757086.913499:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659612800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757086.913524:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757086.913533:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913561:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757086.913568:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757086.913575:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757086.913579:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757086.913583:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757086.913605:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757086.913614:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10141 00000100:00080000:3.0:1787757086.913629:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757086.913640:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757086.913647:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757086.913682:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757086.913696:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757086.913700:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757086.913704:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757086.913709:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757086.913712:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757086.913722:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757086.913733:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757086.913741:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757086.913752:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757086.913764:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913769:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757086.913778:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757086.913788:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913796:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757086.913807:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757086.913812:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913817:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757086.913820:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757086.913823:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913835:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757086.913851:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659612928/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757086.913868:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757086.913873:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.913882:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757086.913888:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757086.913897:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757086.913902:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757086.913905:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757086.913912:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757086.913924:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10141 00000100:00080000:3.0:1787757086.913942:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757086.913950:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757086.913956:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757086.913962:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757086.913967:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757086.913976:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757086.913994:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757086.913998:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757086.914002:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757086.914012:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757086.914022:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757086.914025:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757086.914030:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757086.914035:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914042:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757086.914049:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757086.914060:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914064:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757086.914080:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757086.914085:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914092:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757086.914098:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757086.914101:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914111:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757086.914130:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659613056/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757086.914174:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757086.914231:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914238:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757086.914244:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757086.914251:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757086.914255:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757086.914259:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757086.914266:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757086.914274:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10141 00000100:00080000:3.0:1787757086.914287:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757086.914294:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757086.914298:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757086.914304:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757086.914311:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757086.914318:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757086.914331:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757086.914345:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757086.914353:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757086.914374:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757086.914393:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757086.914405:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757086.914419:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757086.914434:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914440:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757086.914447:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757086.914458:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914467:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757086.914476:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757086.914481:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914489:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757086.914496:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757086.914501:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914509:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757086.914523:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659613184/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757086.914557:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757086.914565:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757086.914577:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757087.681477:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757087.681484:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757087.681490:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757088.705456:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757088.705462:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757088.705468:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757090.753465:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757090.753474:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757090.753480:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757091.777567:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757091.777605:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757091.777613:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757092.033341:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757092.033357:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757092.033363:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757092.033369:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757092.033377:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757092.033389:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10146 00000100:00080000:3.0:1787757092.033406:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757092.033410:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757092.033416:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757092.033422:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757092.033426:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757092.033432:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757092.033435:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757092.033438:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757092.033440:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757092.033448:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757092.033452:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.033456:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757092.033461:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757092.033467:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.033471:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757092.033475:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.033491:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757092.033494:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.033500:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757092.033504:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757092.033512:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.033546:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757092.033564:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659613312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757092.033615:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757092.033625:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.033653:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757092.033659:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757092.033665:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757092.033668:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757092.033671:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757092.033678:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757092.033684:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10146 00000100:00080000:3.0:1787757092.033692:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757092.033696:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757092.033699:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757092.033703:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757092.033706:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757092.033710:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757092.033713:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757092.033716:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757092.033719:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757092.033724:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757092.033728:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.033731:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757092.033740:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757092.033745:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.033746:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757092.033748:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757092.033756:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787757092.033758:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757092.033765:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:3.0:1787757092.033766:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.033773:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757092.033774:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.033781:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.033787:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757092.033791:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757092.033792:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757092.033800:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757092.033800:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757092.033803:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757092.033811:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.033812:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.033828:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757092.033829:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659613312:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757092.033844:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757092.033851:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757092.033855:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757092.033859:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757092.033870:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.033896:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757092.033898:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:2.0:1787757092.033905:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:3.0:1787757092.033947:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659613440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757092.033973:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757092.033990:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757092.033996:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757092.034000:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034003:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757092.034011:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034016:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304819905 00000100:00000001:2.0:1787757092.034031:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757092.034038:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304819905 00000100:00000001:2.0:1787757092.034044:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757092.034049:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757092.034052:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034056:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757092.034066:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757092/real 1787757092] req@ffff8ffb45c31180 x1874588659613312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757092.034081:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757092.034084:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757092.034089:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757092.034092:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757092.034094:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034098:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757092.034102:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757092.034109:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659613312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757092.034119:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757092.034123:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757092.034125:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757092.034132:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034134:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034140:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757092.034143:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757092.034144:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034151:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00080000:3.0:1787757092.034153:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757092.034157:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757092.034164:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757092.034167:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757092.034169:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757092.034179:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757092.034260:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00080000:3.0:1787757092.034269:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757092.034276:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659613312:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757092.034278:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757092.034289:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10146 00000100:00000001:2.0:1787757092.034300:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757092.034301:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757092.034308:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757092.034313:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757092.034314:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757092.034323:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757092.034325:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000020:00000001:3.0:1787757092.034326:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757092.034333:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757092.034335:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000020:00000001:3.0:1787757092.034338:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757092.034343:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787757092.034346:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757092.034346:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757092.034354:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000020:00000001:2.0:1787757092.034355:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757092.034363:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757092.034364:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.034370:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757092.034373:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757092.034376:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757092.034380:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 02000000:00000001:3.0:1787757092.034383:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034389:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757092.034390:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757092.034399:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:3.0:1787757092.034400:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757092.034409:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034411:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757092.034413:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.034424:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757092.034428:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.034431:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757092.034433:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757092.034435:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787757092.034436:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034443:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.034447:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034454:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:3.0:1787757092.034461:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659613568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757092.034462:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757092.034470:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757092.034477:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757092.034483:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757092.034492:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757092.034498:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757092.034499:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757092.034507:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757092.034508:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.034517:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757092.034521:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659613440:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1787757092.034525:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757092.034535:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757092.034537:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 02000000:00000001:2.0:1787757092.034543:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757092.034544:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:2.0:1787757092.034552:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757092.034553:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787757092.034564:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757092.034565:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:2.0:1787757092.034597:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757092.034604:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10146 00000100:00000001:2.0:1787757092.034611:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757092.034615:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757092.034617:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787757092.034625:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757092.034629:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757092.034632:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757092.034640:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757092.034644:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757092.034648:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757092.034654:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000800:00000100:2.0:1787757092.034659:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757092.034663:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757092.034672:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757092.034672:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757092.034679:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757092.034684:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757092.034686:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757092.034688:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757092.034697:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034699:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.034701:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757092.034709:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757092.034712:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304819214 02000000:00000001:3.0:1787757092.034716:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034723:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304819905 00000100:00000001:3.0:1787757092.034723:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757092.034728:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757092.034732:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304819214 00000008:00000001:3.0:1787757092.034735:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034739:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304819905 00000100:00000001:3.0:1787757092.034741:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034750:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757092.034755:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757092.034756:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757092.034761:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034763:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757092.034771:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757092.034777:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:2.0:1787757092.034778:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757092/real 1787757092] req@ffff8ffb45c31500 x1874588659613440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757092.034781:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.034794:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757092.034797:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757092.034802:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:3.0:1787757092.034807:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659613696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757092.034809:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757092.034815:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757092.034820:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034828:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757092.034833:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757092.034834:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757092.034842:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757092.034850:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757092.034851:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659613440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757092.034866:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034869:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757092.034872:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034874:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757092.034876:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034879:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757092.034909:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757092.034914:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757092.034918:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757092.034922:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757092.034926:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659613440:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757092.034931:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034933:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757092.034935:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757092.034936:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757092.034939:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757092.034941:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757092.034943:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757092.034946:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757092.034948:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757092.034951:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757092.034952:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034955:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.034959:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757092.034961:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.034964:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757092.034966:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757092.034968:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757092.034971:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757092.034973:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757092.034976:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757092.034978:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757092.034981:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757092.034985:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659613568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757092.034989:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757092.034991:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757092.034993:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757092.034995:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757092.034999:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035005:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757092.035007:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:2.0:1787757092.035014:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757092.035029:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757092.035034:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757092.035037:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757092.035039:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035041:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757092.035044:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035048:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304819211 00000100:00000001:2.0:1787757092.035051:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304819905 00000100:00000001:2.0:1787757092.035054:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304819211 00000100:00000001:2.0:1787757092.035058:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304819905 00000100:00000001:2.0:1787757092.035060:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757092.035063:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035065:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757092.035070:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757092/real 1787757092] req@ffff8ffb45c33480 x1874588659613568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757092.035079:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757092.035081:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757092.035084:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757092.035086:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757092.035088:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035090:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757092.035092:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757092.035097:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659613568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757092.035105:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035107:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757092.035110:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035113:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757092.035115:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035117:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757092.035136:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757092.035140:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757092.035144:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757092.035147:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757092.035151:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659613568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757092.035155:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757092.035157:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757092.035159:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757092.035161:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757092.035163:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757092.035166:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757092.035169:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659613696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757092.035173:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757092.035175:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757092.035177:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757092.035179:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757092.035183:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035246:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757092.035250:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:2.0:1787757092.035258:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757092.035283:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757092.035295:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757092.035300:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757092.035305:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035309:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757092.035316:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035321:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304819212 00000100:00000001:2.0:1787757092.035325:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304819905 00000100:00000001:2.0:1787757092.035328:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304819212 00000100:00000001:2.0:1787757092.035330:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304819905 00000100:00000001:2.0:1787757092.035333:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757092.035336:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035340:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757092.035346:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757092/real 1787757092] req@ffff8ffb45c31180 x1874588659613696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757092.035357:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757092.035360:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757092.035364:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757092.035367:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757092.035369:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035373:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757092.035376:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757092.035381:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659613696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757092.035396:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035401:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757092.035406:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035409:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757092.035411:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035414:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757092.035448:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757092.035456:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757092.035464:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757092.035469:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757092.035476:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659613696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757092.035483:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035486:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757092.035488:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757092.035490:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757092.035493:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757092.035495:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757092.035498:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757092.035500:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757092.035502:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757092.035506:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757092.035508:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035510:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757092.035511:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757092.035513:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757092.035515:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757092.035517:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757092.035518:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757092.035520:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757092.035521:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757092.035523:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757092.035525:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035527:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757092.035531:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757092.035533:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757092.035535:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757092.035537:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757092.801582:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757092.801589:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757092.801593:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757094.849477:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757094.849483:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757094.849488:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757095.873564:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757095.873572:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757095.873578:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757096.897478:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757096.897524:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757096.897530:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757097.153322:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757097.153335:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757097.153339:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757097.153343:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757097.153349:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757097.153359:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10151 00000100:00080000:3.0:1787757097.153372:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757097.153377:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757097.153380:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757097.153385:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757097.153388:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757097.153392:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757097.153395:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757097.153399:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757097.153401:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757097.153406:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757097.153410:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757097.153414:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757097.153418:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757097.153422:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153425:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757097.153428:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757097.153442:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757097.153444:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153448:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757097.153451:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757097.153457:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153472:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757097.153513:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659613824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757097.153528:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757097.153534:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153558:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757097.153563:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757097.153567:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757097.153569:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757097.153571:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757097.153575:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757097.153579:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10151 00000100:00080000:3.0:1787757097.153587:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757097.153591:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757097.153594:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757097.153611:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757097.153615:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757097.153620:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757097.153627:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757097.153632:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757097.153635:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757097.153644:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757097.153649:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757097.153652:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.153657:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757097.153660:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757097.153663:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757097.153668:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757097.153671:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757097.153673:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757097.153678:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757097.153683:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.153684:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757097.153695:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153695:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757097.153706:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757097.153708:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659613824:192.168.201.118@tcp:250:kworker.0 00000008:00000001:3.0:1787757097.153715:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.153722:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757097.153723:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.153731:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757097.153734:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757097.153737:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757097.153739:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.153744:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757097.153746:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757097.153751:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757097.153755:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757097.153756:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153772:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.153782:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:3.0:1787757097.153783:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659613952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757097.153788:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787757097.153797:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757097.153806:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757097.153810:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153817:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757097.153821:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757097.153826:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757097.153828:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757097.153831:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757097.153835:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757097.153839:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10151 00000100:00080000:3.0:1787757097.153845:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757097.153847:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757097.153851:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:2.0:1787757097.153859:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787757097.153859:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757097.153864:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757097.153869:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757097.153873:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757097.153876:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757097.153879:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757097.153883:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757097.153886:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757097.153892:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.153895:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757097.153896:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757097.153904:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757097.153908:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757097.153913:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757097.153917:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153924:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757097.153928:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757097.153931:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757097.153941:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304825025 00000008:00000001:3.0:1787757097.153941:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153949:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757097.153962:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757097.153963:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757097.153966:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.153974:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757097.153975:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304825025 02000000:00000001:3.0:1787757097.153978:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757097.153986:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.153987:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757097.153996:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757097.154000:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154004:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757097.154006:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757097.154015:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757097/real 1787757097] req@ffff8ffb45c33800 x1874588659613824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757097.154021:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659614080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:2.0:1787757097.154056:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757097.154059:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:2.0:1787757097.154061:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757097.154066:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154071:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757097.154074:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154077:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787757097.154081:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757097.154083:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757097.154105:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757097.154106:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757097.154110:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757097.154112:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757097.154116:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757097.154124:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:2.0:1787757097.154126:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659613824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757097.154137:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10151 00000100:00080000:3.0:1787757097.154149:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757097.154154:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757097.154156:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757097.154161:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757097.154162:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787757097.154167:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757097.154170:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757097.154172:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757097.154177:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:3.0:1787757097.154178:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757097.154181:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757097.154231:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757097.154233:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757097.154240:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757097.154241:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757097.154246:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757097.154249:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757097.154257:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:2.0:1787757097.154263:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757097.154266:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154271:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1787757097.154272:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757097.154280:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:2.0:1787757097.154282:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659613824:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757097.154287:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757097.154293:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757097.154294:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:3.0:1787757097.154298:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757097.154301:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00000001:3.0:1787757097.154306:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154308:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757097.154313:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154314:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757097.154323:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:3.0:1787757097.154325:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000020:00000001:2.0:1787757097.154327:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:3.0:1787757097.154330:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757097.154332:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757097.154335:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757097.154340:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757097.154341:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757097.154345:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154346:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757097.154352:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757097.154356:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757097.154357:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154359:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154374:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00100000:3.0:1787757097.154375:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659614208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757097.154381:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154391:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757097.154395:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757097.154400:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757097.154401:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757097.154408:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757097.154409:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154411:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154418:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757097.154419:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154423:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757097.154428:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757097.154434:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659613952:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757097.154439:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757097.154441:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757097.154443:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154446:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757097.154452:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154460:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757097.154462:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787757097.154468:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757097.154516:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757097.154523:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757097.154527:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757097.154529:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154531:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757097.154535:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154540:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304824334 00000100:00000001:2.0:1787757097.154544:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304825025 00000100:00000001:2.0:1787757097.154548:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304824334 00000100:00000001:2.0:1787757097.154551:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304825025 00000100:00000001:2.0:1787757097.154554:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757097.154556:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154559:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757097.154565:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757097/real 1787757097] req@ffff8ffb45c32a00 x1874588659613952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757097.154575:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757097.154577:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757097.154580:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757097.154582:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757097.154584:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154587:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757097.154589:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757097.154593:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659613952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757097.154602:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154605:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757097.154608:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154611:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757097.154612:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154615:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757097.154631:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757097.154634:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757097.154638:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757097.154641:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757097.154645:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659613952:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757097.154650:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757097.154652:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757097.154655:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154657:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757097.154660:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757097.154662:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757097.154666:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757097.154670:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757097.154672:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757097.154673:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154676:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757097.154679:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154685:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757097.154687:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787757097.154692:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757097.154704:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757097.154709:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757097.154712:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757097.154714:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154716:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757097.154719:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154722:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304824331 00000100:00000001:2.0:1787757097.154726:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304825025 00000100:00000001:2.0:1787757097.154729:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304824331 00000100:00000001:2.0:1787757097.154731:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304825025 00000100:00000001:2.0:1787757097.154734:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757097.154736:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154738:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757097.154743:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757097/real 1787757097] req@ffff8ffb45c30700 x1874588659614080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757097.154752:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757097.154754:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757097.154756:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757097.154758:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757097.154760:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154763:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757097.154765:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757097.154769:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659614080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757097.154778:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154781:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757097.154783:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154785:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757097.154787:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154789:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757097.154802:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757097.154805:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757097.154808:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757097.154811:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757097.154815:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757097.154819:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154821:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757097.154823:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757097.154825:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757097.154828:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757097.154831:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757097.154833:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757097.154838:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757097.154841:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757097.154845:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757097.154847:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154850:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757097.154852:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757097.154855:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757097.154858:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757097.154860:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757097.154862:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757097.154865:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757097.154867:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757097.154870:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757097.154874:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154878:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.154884:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757097.154887:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154891:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757097.154894:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757097.154896:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757097.154899:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757097.154902:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154907:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757097.154910:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757097.154914:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757097.154921:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757097.154928:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757097.154932:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757097.154935:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757097.154938:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757097.154944:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154953:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757097.154955:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787757097.154962:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757097.154983:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757097.154989:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757097.154992:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757097.154994:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.154997:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757097.155000:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.155004:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304824331 00000100:00000001:2.0:1787757097.155008:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304825025 00000100:00000001:2.0:1787757097.155011:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304824331 00000100:00000001:2.0:1787757097.155013:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304825025 00000100:00000001:2.0:1787757097.155016:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757097.155018:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.155021:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757097.155026:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757097/real 1787757097] req@ffff8ffb45c30000 x1874588659614208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757097.155037:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757097.155039:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757097.155041:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757097.155043:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757097.155045:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.155047:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757097.155050:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757097.155054:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659614208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757097.155063:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.155065:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757097.155068:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.155070:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757097.155071:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.155074:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757097.155088:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757097.155091:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757097.155094:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757097.155096:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757097.155100:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757097.155104:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.155106:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757097.155108:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757097.155110:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757097.155112:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757097.155114:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757097.155116:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757097.155118:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757097.155120:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757097.155122:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757097.155124:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.155127:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757097.155131:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757097.155133:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757097.155135:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757097.155138:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757098.945554:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757098.945562:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757098.945570:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757099.969601:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757099.969613:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757099.969620:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787757102.273739:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757102.273756:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757102.273767:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757102.273780:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757102.273784:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757102.273803:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757102.273811:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757102.273823:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757102.273836:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659614336:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757102.273862:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757102.273871:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757102.273878:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757102.273884:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757102.273915:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.273952:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757102.273958:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:2.0:1787757102.273975:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757102.274044:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757102.274061:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757102.274070:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757102.274081:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.274095:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757102.274108:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.274128:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304830145 00000100:00000001:2.0:1787757102.274149:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757102.274172:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304830145 00000100:00000001:2.0:1787757102.274256:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757102.274271:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757102.274289:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.274301:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757102.274319:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757102/real 1787757102] req@ffff8ffb4790ed80 x1874588659614336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757102.274365:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757102.274372:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757102.274410:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757102.274429:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757102.274438:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274453:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757102.274461:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757102.274476:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659614336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757102.274523:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274547:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757102.274568:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274577:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757102.274585:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274602:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757102.274613:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757102.274628:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757102.274644:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757102.274649:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757102.274658:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659614336:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757102.274666:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274670:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757102.274675:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757102.274679:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757102.274687:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757102.274690:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757102.274693:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757102.274700:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757102.274703:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757102.274708:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757102.274711:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274715:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274728:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757102.274732:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.274737:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757102.274740:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757102.274743:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757102.274746:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757102.274748:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757102.274753:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757102.274757:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757102.274762:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757102.274770:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614464:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757102.274778:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757102.274782:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757102.274785:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757102.274789:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757102.274796:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.274808:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757102.274811:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:2.0:1787757102.274819:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757102.274848:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757102.274857:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757102.274862:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757102.274865:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.274869:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757102.274875:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.274883:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304829454 00000100:00000001:2.0:1787757102.274889:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304830145 00000100:00000001:2.0:1787757102.274894:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304829454 00000100:00000001:2.0:1787757102.274899:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304830145 00000100:00000001:2.0:1787757102.274903:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757102.274907:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.274910:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757102.274919:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757102/real 1787757102] req@ffff8ffb4790fb80 x1874588659614464/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757102.274941:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757102.274943:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757102.274947:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757102.274949:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757102.274951:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274953:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757102.274955:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757102.274961:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659614464/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757102.274969:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274973:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757102.274976:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274978:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757102.274979:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.274982:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757102.274997:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757102.275000:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757102.275004:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757102.275007:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757102.275011:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614464:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757102.275016:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757102.275019:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757102.275021:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757102.275024:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757102.275027:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757102.275030:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757102.275033:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757102.275038:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757102.275041:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757102.275044:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757102.275046:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757102.275051:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275058:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757102.275060:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:2.0:1787757102.275066:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757102.275082:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757102.275088:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757102.275091:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757102.275093:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275095:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757102.275098:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275102:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304829451 00000100:00000001:2.0:1787757102.275106:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304830145 00000100:00000001:2.0:1787757102.275109:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304829451 00000100:00000001:2.0:1787757102.275112:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304830145 00000100:00000001:2.0:1787757102.275114:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757102.275117:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275119:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757102.275124:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757102/real 1787757102] req@ffff8ffb4790d180 x1874588659614592/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757102.275133:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757102.275134:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757102.275137:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757102.275139:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757102.275141:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275143:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757102.275145:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757102.275149:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659614592/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757102.275158:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275160:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757102.275164:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275166:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757102.275168:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275170:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757102.275211:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757102.275215:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757102.275220:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757102.275225:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757102.275232:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757102.275238:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757102.275240:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757102.275243:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757102.275246:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757102.275248:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757102.275251:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757102.275255:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757102.275260:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757102.275262:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757102.275264:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757102.275266:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757102.275270:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275276:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757102.275278:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:2.0:1787757102.275283:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757102.275296:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757102.275302:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757102.275304:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757102.275307:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275309:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757102.275312:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275317:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304829452 00000100:00000001:2.0:1787757102.275320:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304830145 00000100:00000001:2.0:1787757102.275324:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304829452 00000100:00000001:2.0:1787757102.275327:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304830145 00000100:00000001:2.0:1787757102.275329:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757102.275332:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275334:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757102.275339:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757102/real 1787757102] req@ffff8ffb4790e680 x1874588659614720/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757102.275348:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757102.275350:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757102.275352:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757102.275354:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757102.275356:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275359:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757102.275360:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757102.275365:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659614720/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757102.275373:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275376:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757102.275378:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275380:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757102.275382:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275384:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757102.275420:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757102.275425:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757102.275429:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757102.275432:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757102.275436:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757102.275441:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275443:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757102.275446:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757102.275448:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757102.275451:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757102.275453:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757102.275455:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757102.275459:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757102.275461:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757102.275464:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757102.275465:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275467:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757102.275469:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757102.275471:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757102.275473:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757102.275474:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757102.275476:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757102.275477:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757102.275479:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757102.275480:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757102.275482:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275484:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757102.275485:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757102.275486:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757102.275488:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757102.275489:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757102.275491:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757102.275493:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757102.275494:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757102.275495:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757102.275497:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275499:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757102.275504:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757102.275505:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757102.275508:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757102.275510:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757103.041531:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757103.041538:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757103.041544:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757104.065505:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757104.065513:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757104.065520:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757107.137511:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757107.137532:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757107.137539:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787757107.393659:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757107.393670:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757107.393679:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757107.393684:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757107.393690:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757107.393699:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757107.393706:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757107.393736:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757107.393752:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659614848:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757107.393768:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757107.393778:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757107.393784:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757107.393792:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757107.393802:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.393834:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757107.393842:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:2.0:1787757107.393858:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757107.393931:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757107.393970:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757107.393985:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757107.393992:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394018:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757107.394030:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394041:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304835265 00000100:00000001:2.0:1787757107.394076:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757107.394091:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304835265 00000100:00000001:2.0:1787757107.394103:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757107.394114:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757107.394125:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394133:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757107.394159:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757107/real 1787757107] req@ffff8ffb4790d880 x1874588659614848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757107.394239:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757107.394250:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757107.394264:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757107.394269:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757107.394271:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394275:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757107.394278:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757107.394287:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659614848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757107.394345:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394349:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757107.394354:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394359:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757107.394361:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394366:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757107.394372:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757107.394381:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757107.394398:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757107.394410:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757107.394426:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659614848:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757107.394442:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394456:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757107.394465:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757107.394474:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757107.394503:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757107.394514:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757107.394524:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757107.394535:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757107.394542:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757107.394551:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757107.394558:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394570:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394591:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757107.394598:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394604:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757107.394607:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757107.394611:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757107.394614:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757107.394618:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757107.394640:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757107.394645:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757107.394649:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757107.394659:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614976:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757107.394666:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757107.394670:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757107.394673:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757107.394676:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757107.394682:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394693:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757107.394694:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:2.0:1787757107.394700:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757107.394720:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757107.394727:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757107.394747:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757107.394749:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394751:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757107.394756:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394762:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304834574 00000100:00000001:2.0:1787757107.394766:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304835265 00000100:00000001:2.0:1787757107.394769:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304834574 00000100:00000001:2.0:1787757107.394772:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304835265 00000100:00000001:2.0:1787757107.394775:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757107.394778:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394780:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757107.394786:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757107/real 1787757107] req@ffff8ffb4790f100 x1874588659614976/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757107.394798:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757107.394799:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757107.394803:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757107.394805:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757107.394807:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394810:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757107.394812:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757107.394817:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659614976/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757107.394834:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394837:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757107.394841:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394843:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757107.394844:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.394846:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757107.394861:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757107.394864:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757107.394868:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757107.394870:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757107.394875:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659614976:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757107.394880:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757107.394882:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757107.394884:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757107.394886:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757107.394889:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757107.394892:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757107.394895:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757107.394899:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757107.394901:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757107.394903:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757107.394905:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757107.394909:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394916:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757107.394917:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:2.0:1787757107.394922:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757107.394954:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757107.394958:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757107.394961:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757107.394963:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394965:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757107.394969:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394972:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304834571 00000100:00000001:2.0:1787757107.394975:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304835265 00000100:00000001:2.0:1787757107.394978:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304834571 00000100:00000001:2.0:1787757107.394980:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304835265 00000100:00000001:2.0:1787757107.394983:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757107.394985:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.394987:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757107.394991:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757107/real 1787757107] req@ffff8ffb4790c380 x1874588659615104/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757107.395000:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757107.395001:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757107.395004:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757107.395006:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757107.395007:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395009:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757107.395011:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757107.395015:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659615104/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757107.395023:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395026:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757107.395028:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395030:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757107.395031:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395033:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757107.395044:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757107.395047:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757107.395050:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757107.395052:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757107.395057:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757107.395061:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757107.395063:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757107.395065:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757107.395067:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757107.395069:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757107.395071:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757107.395074:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757107.395078:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757107.395080:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757107.395081:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757107.395084:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757107.395087:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.395092:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757107.395093:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:2.0:1787757107.395097:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757107.395108:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757107.395112:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757107.395114:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757107.395116:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.395118:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757107.395121:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.395125:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304834571 00000100:00000001:2.0:1787757107.395128:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304835265 00000100:00000001:2.0:1787757107.395130:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304834571 00000100:00000001:2.0:1787757107.395133:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304835265 00000100:00000001:2.0:1787757107.395135:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757107.395137:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.395139:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757107.395143:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757107/real 1787757107] req@ffff8ffb4790ce00 x1874588659615232/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757107.395160:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757107.395161:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757107.395164:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757107.395166:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757107.395167:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395169:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757107.395170:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757107.395174:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659615232/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757107.395182:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395184:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757107.395187:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395189:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757107.395191:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395193:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757107.395204:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757107.395207:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757107.395210:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757107.395212:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757107.395216:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757107.395224:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395229:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757107.395249:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757107.395253:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757107.395257:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757107.395343:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757107.395347:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757107.395351:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757107.395353:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757107.395356:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757107.395358:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395361:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757107.395362:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757107.395364:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757107.395366:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757107.395368:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757107.395369:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757107.395372:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757107.395373:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757107.395375:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757107.395376:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395378:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757107.395379:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757107.395380:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757107.395382:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757107.395383:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757107.395384:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757107.395386:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757107.395387:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757107.395389:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757107.395390:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395392:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757107.395397:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757107.395399:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757107.395401:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757107.395403:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757108.161478:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757108.161486:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757108.161492:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757111.233432:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757111.233439:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757111.233445:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757112.257503:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757112.257512:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757112.257519:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787757112.513711:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757112.513721:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757112.513728:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757112.513734:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757112.513742:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757112.513749:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757112.513755:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757112.513765:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757112.513781:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659615360:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757112.513797:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757112.513806:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757112.513812:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757112.513818:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757112.513830:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.513858:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757112.513865:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133081856 : -123165576469760 : ffff8ffb4790e300) 00000100:00000001:2.0:1787757112.513874:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757112.513957:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757112.514007:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757112.514021:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757112.514029:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514035:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757112.514053:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514067:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304840385 00000100:00000001:2.0:1787757112.514095:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757112.514108:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304840385 00000100:00000001:2.0:1787757112.514121:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757112.514156:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757112.514168:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514178:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757112.514274:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757112/real 1787757112] req@ffff8ffb4790e300 x1874588659615360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757112.514302:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757112.514307:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757112.514316:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757112.514323:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757112.514328:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514338:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757112.514345:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757112.514358:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e300 x1874588659615360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757112.514378:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514384:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757112.514389:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514391:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757112.514393:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514397:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757112.514402:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757112.514407:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757112.514414:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757112.514418:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757112.514434:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659615360:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757112.514442:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514445:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757112.514448:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757112.514452:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757112.514457:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757112.514461:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757112.514464:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757112.514471:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757112.514473:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757112.514477:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757112.514478:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514482:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514492:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757112.514495:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514498:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757112.514500:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757112.514502:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757112.514505:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757112.514507:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757112.514510:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757112.514513:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757112.514516:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757112.514520:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615488:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757112.514524:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757112.514527:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757112.514529:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757112.514532:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757112.514536:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514544:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757112.514546:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133087232 : -123165576464384 : ffff8ffb4790f800) 00000100:00000001:2.0:1787757112.514553:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757112.514574:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757112.514581:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757112.514584:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757112.514586:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514588:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757112.514592:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514598:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304839694 00000100:00000001:2.0:1787757112.514602:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304840385 00000100:00000001:2.0:1787757112.514605:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304839694 00000100:00000001:2.0:1787757112.514608:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304840385 00000100:00000001:2.0:1787757112.514611:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757112.514614:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514616:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757112.514621:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757112/real 1787757112] req@ffff8ffb4790f800 x1874588659615488/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757112.514631:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757112.514633:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757112.514635:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757112.514638:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757112.514640:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514642:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757112.514644:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757112.514649:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f800 x1874588659615488/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757112.514658:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514660:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757112.514663:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514665:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757112.514667:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514669:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757112.514690:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757112.514693:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757112.514698:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757112.514701:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757112.514706:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615488:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757112.514711:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757112.514712:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757112.514714:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757112.514717:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757112.514719:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757112.514721:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757112.514724:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757112.514728:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757112.514730:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757112.514732:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757112.514734:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757112.514738:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514745:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757112.514747:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133078272 : -123165576473344 : ffff8ffb4790d500) 00000100:00000001:2.0:1787757112.514752:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757112.514764:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757112.514769:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757112.514772:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757112.514774:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514776:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757112.514779:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514784:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304839691 00000100:00000001:2.0:1787757112.514787:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304840385 00000100:00000001:2.0:1787757112.514790:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304839691 00000100:00000001:2.0:1787757112.514793:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304840385 00000100:00000001:2.0:1787757112.514796:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757112.514798:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514800:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757112.514806:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757112/real 1787757112] req@ffff8ffb4790d500 x1874588659615616/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757112.514816:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757112.514818:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757112.514821:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757112.514822:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757112.514824:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514826:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757112.514828:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757112.514832:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d500 x1874588659615616/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757112.514842:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514844:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757112.514847:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514849:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757112.514850:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.514852:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757112.514871:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757112.514874:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757112.514877:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757112.514880:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757112.514883:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757112.514887:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757112.514889:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757112.514891:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757112.514894:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757112.514896:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757112.514899:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757112.514902:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757112.514905:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757112.514908:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757112.514910:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757112.514913:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757112.514918:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514926:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757112.514929:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:2.0:1787757112.514936:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757112.514957:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757112.514966:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757112.514970:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757112.514973:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514977:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757112.514982:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.514988:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304839691 00000100:00000001:2.0:1787757112.514994:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304840385 00000100:00000001:2.0:1787757112.514999:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304839691 00000100:00000001:2.0:1787757112.515004:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304840385 00000100:00000001:2.0:1787757112.515008:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757112.515013:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.515016:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757112.515026:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757112/real 1787757112] req@ffff8ffb4790ce00 x1874588659615744/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757112.515040:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757112.515042:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757112.515046:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757112.515048:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757112.515050:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515052:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757112.515054:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757112.515059:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659615744/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757112.515068:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515071:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757112.515075:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515077:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757112.515079:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515082:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757112.515101:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757112.515104:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757112.515108:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757112.515111:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757112.515115:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659615744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757112.515119:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515122:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757112.515123:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757112.515126:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757112.515128:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757112.515130:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757112.515132:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757112.515137:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757112.515139:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757112.515142:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757112.515144:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515146:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757112.515148:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757112.515149:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757112.515152:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757112.515153:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757112.515155:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757112.515156:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757112.515158:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757112.515160:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757112.515161:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515163:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757112.515164:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757112.515166:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757112.515167:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757112.515168:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757112.515170:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757112.515171:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757112.515174:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757112.515176:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757112.515178:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515180:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757112.515242:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757112.515245:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757112.515248:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757112.515251:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757115.329583:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757115.329591:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757115.329598:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757116.353495:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757116.353503:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757116.353508:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787757117.633721:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757117.633730:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757117.633738:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757117.633746:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757117.633753:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757117.633763:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757117.633769:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757117.633776:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757117.633791:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659615872:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757117.633810:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757117.633822:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757117.633828:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757117.633835:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757117.633848:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.633882:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757117.633890:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:2.0:1787757117.633900:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757117.633969:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757117.633994:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757117.634023:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757117.634032:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634040:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757117.634056:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634068:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304845505 00000100:00000001:2.0:1787757117.634092:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757117.634107:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304845505 00000100:00000001:2.0:1787757117.634162:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757117.634229:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757117.634237:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634250:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757117.634264:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757117/real 1787757117] req@ffff8ffb4790c380 x1874588659615872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757117.634310:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757117.634315:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757117.634326:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757117.634335:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757117.634340:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634345:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757117.634348:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757117.634357:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659615872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757117.634369:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634374:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757117.634378:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634380:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757117.634382:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634385:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757117.634388:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757117.634392:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757117.634397:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757117.634400:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757117.634405:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659615872:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757117.634412:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634414:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757117.634416:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757117.634420:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757117.634425:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757117.634428:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757117.634431:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757117.634438:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757117.634441:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757117.634445:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757117.634447:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634450:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634461:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757117.634464:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634466:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757117.634468:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757117.634471:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757117.634473:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757117.634475:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757117.634478:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757117.634481:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757117.634484:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757117.634488:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616000:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757117.634492:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757117.634495:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757117.634497:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757117.634499:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757117.634503:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634510:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757117.634512:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:2.0:1787757117.634518:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757117.634536:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757117.634542:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757117.634545:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757117.634547:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634549:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757117.634553:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634559:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304844814 00000100:00000001:2.0:1787757117.634564:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304845505 00000100:00000001:2.0:1787757117.634567:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304844814 00000100:00000001:2.0:1787757117.634570:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304845505 00000100:00000001:2.0:1787757117.634573:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757117.634575:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634579:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757117.634584:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757117/real 1787757117] req@ffff8ffb4790f100 x1874588659616000/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757117.634596:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757117.634597:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757117.634600:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757117.634602:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757117.634604:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634607:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757117.634609:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757117.634614:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659616000/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757117.634622:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634624:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757117.634627:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634629:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757117.634631:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634633:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757117.634655:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757117.634659:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757117.634663:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757117.634666:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757117.634670:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616000:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757117.634675:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757117.634677:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757117.634678:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757117.634681:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757117.634683:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757117.634685:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757117.634689:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616128:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757117.634697:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757117.634699:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757117.634701:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757117.634703:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757117.634707:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634713:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757117.634715:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:2.0:1787757117.634720:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757117.634732:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757117.634737:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757117.634739:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757117.634741:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634743:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757117.634746:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634750:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304844811 00000100:00000001:2.0:1787757117.634754:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304845505 00000100:00000001:2.0:1787757117.634757:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304844811 00000100:00000001:2.0:1787757117.634760:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304845505 00000100:00000001:2.0:1787757117.634762:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757117.634765:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634767:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757117.634771:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757117/real 1787757117] req@ffff8ffb4790d880 x1874588659616128/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757117.634780:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757117.634781:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757117.634784:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757117.634786:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757117.634787:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634789:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757117.634791:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757117.634795:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659616128/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757117.634811:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634814:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757117.634817:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634819:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757117.634821:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634823:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757117.634841:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757117.634844:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757117.634847:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757117.634849:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757117.634853:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616128:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757117.634858:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757117.634860:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757117.634861:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757117.634864:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757117.634867:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757117.634869:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757117.634872:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757117.634876:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757117.634877:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757117.634879:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757117.634881:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757117.634884:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634889:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757117.634891:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:2.0:1787757117.634895:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757117.634906:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757117.634910:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757117.634913:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757117.634914:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634917:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757117.634920:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634924:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304844811 00000100:00000001:2.0:1787757117.634927:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304845505 00000100:00000001:2.0:1787757117.634930:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304844811 00000100:00000001:2.0:1787757117.634933:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304845505 00000100:00000001:2.0:1787757117.634936:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757117.634938:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.634940:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757117.634945:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757117/real 1787757117] req@ffff8ffb4790e680 x1874588659616256/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757117.634953:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757117.634955:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757117.634957:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757117.634959:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757117.634961:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634963:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757117.634965:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757117.634970:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659616256/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757117.634978:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634980:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757117.634983:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634984:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757117.634986:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.634988:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757117.635007:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757117.635010:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757117.635014:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757117.635017:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757117.635022:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757117.635027:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.635029:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757117.635031:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757117.635033:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757117.635035:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757117.635037:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757117.635038:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757117.635042:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757117.635043:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757117.635046:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757117.635048:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.635050:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757117.635051:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757117.635053:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757117.635055:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757117.635057:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757117.635058:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757117.635061:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757117.635062:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757117.635064:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757117.635066:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.635067:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757117.635069:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757117.635070:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757117.635072:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757117.635073:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757117.635074:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757117.635076:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757117.635077:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757117.635080:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757117.635081:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.635083:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757117.635087:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757117.635088:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757117.635091:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757117.635093:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757119.425527:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757119.425535:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757119.425543:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757120.449404:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757120.449411:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757120.449416:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787757122.753636:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757122.753646:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757122.753654:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757122.753661:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757122.753667:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757122.753679:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757122.753687:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757122.753694:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757122.753709:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659616384:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757122.753722:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757122.753733:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757122.753741:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757122.753752:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757122.753763:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.753792:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757122.753797:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:2.0:1787757122.753804:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757122.753875:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757122.753897:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757122.753906:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757122.753915:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.753924:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757122.753937:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.753948:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304850625 00000100:00000001:2.0:1787757122.753968:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757122.753984:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304850625 00000100:00000001:2.0:1787757122.754018:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757122.754067:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757122.754078:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754088:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757122.754102:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757122/real 1787757122] req@ffff8ffb4790d180 x1874588659616384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757122.754139:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757122.754145:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757122.754165:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757122.754172:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757122.754240:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754248:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757122.754252:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757122.754264:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659616384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757122.754282:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754289:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757122.754295:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754298:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757122.754300:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754304:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757122.754307:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757122.754311:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757122.754317:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757122.754321:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757122.754326:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659616384:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757122.754333:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754337:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757122.754339:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757122.754343:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757122.754349:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757122.754352:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757122.754355:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757122.754363:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757122.754365:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757122.754369:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757122.754371:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754374:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754385:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757122.754388:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754391:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757122.754393:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757122.754395:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757122.754397:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757122.754400:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757122.754403:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757122.754406:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757122.754409:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757122.754413:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616512:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757122.754418:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757122.754421:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757122.754423:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757122.754425:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757122.754430:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754438:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757122.754440:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:2.0:1787757122.754447:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757122.754466:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757122.754473:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757122.754476:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757122.754478:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754480:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757122.754484:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754490:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304849934 00000100:00000001:2.0:1787757122.754494:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304850625 00000100:00000001:2.0:1787757122.754497:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304849934 00000100:00000001:2.0:1787757122.754500:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304850625 00000100:00000001:2.0:1787757122.754503:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757122.754505:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754508:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757122.754513:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757122/real 1787757122] req@ffff8ffb4790fb80 x1874588659616512/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757122.754523:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757122.754525:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757122.754528:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757122.754530:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757122.754532:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754534:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757122.754536:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757122.754541:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659616512/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757122.754551:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754553:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757122.754556:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754558:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757122.754560:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754562:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757122.754583:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757122.754586:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757122.754590:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757122.754593:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757122.754597:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616512:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757122.754601:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757122.754603:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757122.754605:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757122.754608:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757122.754610:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757122.754613:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757122.754616:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616640:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757122.754619:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757122.754621:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757122.754623:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757122.754625:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757122.754628:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754635:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757122.754636:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:2.0:1787757122.754642:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757122.754654:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757122.754658:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757122.754661:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757122.754664:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754666:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757122.754669:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754673:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304849931 00000100:00000001:2.0:1787757122.754676:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304850625 00000100:00000001:2.0:1787757122.754679:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304849931 00000100:00000001:2.0:1787757122.754682:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304850625 00000100:00000001:2.0:1787757122.754685:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757122.754687:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754689:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757122.754693:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757122/real 1787757122] req@ffff8ffb4790ed80 x1874588659616640/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757122.754702:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757122.754704:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757122.754707:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757122.754708:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757122.754710:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754713:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757122.754714:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757122.754719:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659616640/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757122.754727:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754729:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757122.754732:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754734:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757122.754735:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754737:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757122.754755:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757122.754758:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757122.754761:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757122.754764:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757122.754768:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616640:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757122.754773:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757122.754775:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757122.754776:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757122.754779:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757122.754781:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757122.754783:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757122.754786:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757122.754789:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757122.754791:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757122.754793:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757122.754795:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757122.754798:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754803:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757122.754805:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:2.0:1787757122.754809:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757122.754820:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757122.754825:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757122.754827:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757122.754829:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754832:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757122.754834:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754838:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304849931 00000100:00000001:2.0:1787757122.754842:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304850625 00000100:00000001:2.0:1787757122.754845:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304849931 00000100:00000001:2.0:1787757122.754847:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304850625 00000100:00000001:2.0:1787757122.754850:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757122.754852:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.754854:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757122.754859:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757122/real 1787757122] req@ffff8ffb4790ca80 x1874588659616768/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757122.754877:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757122.754878:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757122.754881:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757122.754883:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757122.754885:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754887:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757122.754888:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757122.754893:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659616768/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757122.754900:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754903:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757122.754905:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754907:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757122.754909:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754911:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757122.754929:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757122.754932:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757122.754935:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757122.754938:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757122.754942:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659616768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757122.754947:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754949:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757122.754951:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757122.754952:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757122.754955:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757122.754956:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757122.754958:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757122.754961:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757122.754963:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757122.754965:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757122.754967:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754969:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757122.754971:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757122.754972:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757122.754975:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757122.754977:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757122.754978:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757122.754980:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757122.754981:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757122.754984:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757122.754985:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.754987:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757122.754988:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757122.754990:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757122.754991:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757122.754993:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757122.754994:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757122.754995:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757122.754997:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757122.754999:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757122.755000:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.755002:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757122.755006:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757122.755008:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757122.755010:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757122.755012:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757123.521481:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757123.521490:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757123.521497:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757124.545504:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757124.545511:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757124.545516:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757127.617368:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757127.617373:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757127.617378:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757127.873357:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757127.873385:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757127.873389:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757127.873395:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757127.873402:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757127.873412:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10182 00000100:00080000:3.0:1787757127.873427:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757127.873432:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757127.873435:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757127.873441:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757127.873445:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757127.873449:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757127.873453:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757127.873456:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757127.873458:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757127.873464:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757127.873468:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757127.873472:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757127.873477:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757127.873481:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.873484:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757127.873487:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757127.873501:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757127.873503:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.873508:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757127.873512:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757127.873517:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.873536:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757127.873549:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659616896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757127.873563:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757127.873568:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.873595:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757127.873601:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757127.873606:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757127.873608:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757127.873611:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757127.873615:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757127.873621:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10182 00000100:00080000:3.0:1787757127.873627:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757127.873630:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757127.873633:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757127.873636:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757127.873639:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757127.873653:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757127.873658:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757127.873663:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757127.873668:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757127.873697:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757127.873710:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757127.873721:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757127.873737:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757127.873746:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.873772:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757127.873785:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757127.873807:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.873813:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757127.873977:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757127.873981:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.873984:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757127.873986:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757127.873990:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.873998:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757127.874031:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659617024/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757127.874315:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757127.874319:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875469:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757127.875480:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757127.875487:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757127.875490:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757127.875494:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757127.875500:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757127.875506:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10182 00000100:00080000:3.0:1787757127.875516:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757127.875520:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757127.875523:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757127.875528:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757127.875530:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757127.875535:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757127.875539:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757127.875543:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757127.875545:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757127.875551:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757127.875557:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757127.875560:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757127.875568:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757127.875573:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875578:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757127.875583:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757127.875587:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875590:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757127.875603:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757127.875606:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875610:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757127.875613:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757127.875617:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875628:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757127.875639:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659617152/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757127.875653:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757127.875657:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875692:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757127.875698:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757127.875704:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757127.875706:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757127.875710:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757127.875714:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757127.875719:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10182 00000100:00080000:3.0:1787757127.875727:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757127.875732:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757127.875735:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757127.875740:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757127.875743:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757127.875747:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757127.875759:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757127.875764:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757127.875766:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757127.875772:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757127.875779:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757127.875783:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757127.875789:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757127.875794:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875797:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757127.875801:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757127.875805:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875809:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757127.875818:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757127.875824:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875830:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757127.875837:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757127.875847:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875862:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757127.875879:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659617280/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757127.875909:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757127.875958:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757127.875979:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757128.641459:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757128.641465:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757128.641471:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757131.713457:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757131.713463:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757131.713468:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757132.737636:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757132.737642:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757132.737648:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757132.993482:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757132.993498:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757132.993503:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757132.993508:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757132.993515:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757132.993526:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10187 00000100:00080000:3.0:1787757132.993543:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757132.993549:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757132.993564:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757132.993573:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757132.993578:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757132.993584:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757132.993590:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757132.993594:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757132.993596:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757132.993605:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757132.993612:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757132.993617:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757132.993624:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757132.993630:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.993634:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757132.993639:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757132.993669:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757132.993673:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.993681:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757132.993686:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757132.993695:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.993729:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757132.993750:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659617408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757132.993772:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757132.993781:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.993817:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757132.993824:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757132.993832:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757132.993835:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757132.993838:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757132.993878:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757132.993886:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10187 00000100:00080000:3.0:1787757132.993894:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757132.993898:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757132.993901:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757132.993905:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757132.993908:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757132.993912:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757132.993916:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757132.993920:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757132.993923:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757132.993929:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757132.993935:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757132.993939:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757132.993949:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757132.993954:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.993958:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757132.993966:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757132.993972:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.993975:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757132.993986:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757132.993989:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.993993:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757132.993997:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757132.994001:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.994012:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757132.994024:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659617536/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757132.994072:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757132.994077:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.994085:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757132.994096:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757132.994109:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757132.994116:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757132.994122:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757132.994138:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757132.994152:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10187 00000100:00080000:3.0:1787757132.994168:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757132.994179:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757132.994228:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757132.994234:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757132.994237:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757132.994242:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757132.994247:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757132.994252:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757132.994256:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757132.994264:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757132.994274:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757132.994282:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757132.994289:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757132.994294:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.994298:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757132.994304:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757132.994314:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.994323:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757132.994334:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757132.994338:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.994342:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757132.994346:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757132.994356:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.994376:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757132.994392:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659617664/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757132.995264:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757132.995279:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.995328:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757132.995342:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757132.995356:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757132.995363:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757132.995373:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757132.995386:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757132.995416:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10187 00000100:00080000:3.0:1787757132.995432:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757132.995442:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757132.995451:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757132.995462:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757132.995469:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757132.995478:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757132.995488:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757132.995518:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757132.995527:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757132.995545:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757132.995555:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757132.995561:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757132.995573:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757132.995582:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.995590:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757132.995595:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757132.995602:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.995607:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757132.995639:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757132.995646:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.995655:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757132.995663:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757132.995672:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.995690:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757132.995712:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659617792/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757132.995752:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757132.995760:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757132.995772:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757135.809449:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757135.809465:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757135.809470:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757136.833335:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757136.833338:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757136.833340:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757139.905550:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757139.905557:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757139.905562:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757140.929445:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757140.929451:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757140.929455:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757143.233370:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757143.233387:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757143.233391:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757143.233397:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757143.233404:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757143.233415:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10197 00000100:00080000:3.0:1787757143.233431:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757143.233435:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757143.233439:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757143.233445:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757143.233449:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757143.233454:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757143.233458:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757143.233461:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757143.233463:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757143.233470:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757143.233475:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757143.233493:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757143.233500:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757143.233508:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233512:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757143.233517:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757143.233541:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757143.233545:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233552:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757143.233558:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757143.233566:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233592:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757143.233610:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659618432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757143.233626:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757143.233633:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233660:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757143.233708:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757143.233715:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757143.233718:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757143.233721:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757143.233726:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757143.233732:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10197 00000100:00080000:3.0:1787757143.233740:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757143.233749:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757143.233754:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757143.233762:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757143.233769:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757143.233774:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757143.233780:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757143.233786:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757143.233790:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757143.233800:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757143.233807:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757143.233812:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757143.233824:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757143.233831:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233837:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757143.233845:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757143.233854:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233859:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757143.233871:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757143.233876:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233881:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757143.233883:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757143.233886:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233897:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757143.233908:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659618560/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757143.233947:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757143.233952:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.233957:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757143.233961:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757143.233967:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757143.233970:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757143.233973:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757143.233977:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757143.233981:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10197 00000100:00080000:3.0:1787757143.233986:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757143.233993:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757143.233997:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757143.234005:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757143.234007:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757143.234012:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757143.234015:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757143.234018:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757143.234020:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757143.234027:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757143.234034:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757143.234046:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757143.234053:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757143.234057:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234063:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757143.234067:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757143.234076:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234078:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757143.234086:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757143.234089:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234097:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757143.234099:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757143.234105:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234124:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757143.234143:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659618688/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757143.234163:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757143.234169:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234177:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757143.234241:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757143.234253:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757143.234258:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757143.234265:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757143.234274:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757143.234284:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10197 00000100:00080000:3.0:1787757143.234293:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757143.234302:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757143.234307:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757143.234317:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757143.234321:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757143.234327:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757143.234331:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757143.234333:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757143.234334:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757143.234340:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757143.234343:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757143.234348:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757143.234355:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757143.234362:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234369:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757143.234376:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757143.234385:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234392:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757143.234402:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757143.234408:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234417:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757143.234419:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757143.234426:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234439:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757143.234457:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659618816/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757143.234485:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757143.234494:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757143.234506:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757144.001440:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757144.001447:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757144.001452:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757145.025418:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757145.025424:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757145.025430:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757148.097437:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757148.097444:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757148.097449:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757148.353313:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757148.353330:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757148.353334:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757148.353339:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757148.353347:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757148.353358:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10203 00000100:00080000:3.0:1787757148.353391:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757148.353396:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757148.353399:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757148.353405:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757148.353409:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757148.353413:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757148.353417:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757148.353419:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757148.353421:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757148.353428:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757148.353431:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757148.353435:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757148.353439:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757148.353461:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353464:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757148.353467:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757148.353488:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757148.353490:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353495:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757148.353500:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757148.353508:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353527:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757148.353542:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659618944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757148.353559:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757148.353565:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353632:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757148.353637:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757148.353642:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757148.353644:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757148.353646:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757148.353650:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757148.353654:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10203 00000100:00080000:3.0:1787757148.353659:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757148.353662:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757148.353664:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757148.353668:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757148.353670:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757148.353674:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757148.353677:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757148.353680:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757148.353681:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757148.353685:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757148.353689:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757148.353693:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757148.353703:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757148.353711:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353718:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757148.353729:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757148.353736:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353742:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757148.353751:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757148.353756:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353765:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757148.353768:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757148.353774:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353785:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757148.353797:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659619072/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757148.353816:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757148.353823:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353828:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757148.353832:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757148.353837:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757148.353839:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757148.353841:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757148.353845:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757148.353848:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10203 00000100:00080000:3.0:1787757148.353852:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757148.353860:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757148.353862:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757148.353870:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757148.353875:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757148.353879:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757148.353887:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757148.353890:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757148.353893:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757148.353897:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757148.353899:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757148.353901:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757148.353904:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757148.353906:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353908:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757148.353912:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757148.353934:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353937:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757148.353943:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757148.353945:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353948:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757148.353950:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757148.353952:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353959:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757148.353965:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659619200/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757148.353975:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757148.353977:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.353980:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757148.353984:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757148.354006:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757148.354011:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757148.354015:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757148.354022:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757148.354030:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10203 00000100:00080000:3.0:1787757148.354045:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757148.354057:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757148.354065:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757148.354077:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757148.354084:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757148.354092:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757148.354100:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757148.354111:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757148.354115:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757148.354128:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757148.354139:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757148.354146:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757148.354154:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757148.354163:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.354168:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757148.354175:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757148.354219:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.354224:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757148.354232:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757148.354234:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.354238:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757148.354240:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757148.354242:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.354250:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757148.354266:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659619328/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757148.354282:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757148.354292:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757148.354301:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757149.121467:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757149.121473:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757149.121478:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757152.193400:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757152.193405:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757152.193409:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757153.217437:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757153.217444:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757153.217450:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757153.473325:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757153.473342:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757153.473345:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757153.473352:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757153.473359:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757153.473370:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10208 00000100:00080000:3.0:1787757153.473388:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757153.473395:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757153.473398:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757153.473404:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757153.473408:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757153.473413:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757153.473416:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757153.473419:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757153.473421:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757153.473428:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757153.473431:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757153.473435:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757153.473440:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757153.473445:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473448:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757153.473451:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757153.473476:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757153.473480:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473542:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757153.473547:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757153.473557:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473583:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757153.473604:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659619456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757153.473624:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757153.473630:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473655:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757153.473660:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757153.473665:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757153.473668:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757153.473670:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757153.473674:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757153.473678:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10208 00000100:00080000:3.0:1787757153.473685:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757153.473689:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757153.473691:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757153.473695:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757153.473698:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757153.473702:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757153.473705:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757153.473708:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757153.473710:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757153.473716:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757153.473724:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757153.473726:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757153.473735:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757153.473739:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473748:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757153.473759:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757153.473770:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473777:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757153.473787:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757153.473792:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473800:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757153.473805:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757153.473810:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473824:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757153.473839:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659619584/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757153.473861:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757153.473866:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473874:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757153.473878:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757153.473883:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757153.473886:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757153.473889:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757153.473893:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757153.473897:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10208 00000100:00080000:3.0:1787757153.473903:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757153.473910:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757153.473916:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757153.473922:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757153.473927:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757153.473930:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757153.473934:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757153.473937:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757153.473939:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757153.473943:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757153.473947:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757153.473949:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757153.473953:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757153.473956:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473958:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757153.473962:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757153.473965:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473968:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757153.473975:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757153.473976:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473979:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757153.473980:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757153.473983:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.473991:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757153.474003:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659619712/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757153.474015:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757153.474019:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.474030:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757153.474039:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757153.474050:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757153.474054:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757153.474057:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757153.474062:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757153.474070:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10208 00000100:00080000:3.0:1787757153.474076:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757153.474081:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757153.474083:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757153.474085:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757153.474087:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757153.474089:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757153.474093:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757153.474101:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757153.474104:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757153.474111:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757153.474114:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757153.474116:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757153.474120:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757153.474122:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.474127:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757153.474133:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757153.474136:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.474139:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757153.474145:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757153.474152:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.474155:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757153.474158:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757153.474163:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.474172:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757153.474220:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659619840/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757153.474232:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757153.474236:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757153.474241:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757156.289493:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757156.289500:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757156.289505:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757157.313483:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757157.313490:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757157.313495:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757160.385388:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757160.385394:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757160.385398:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757161.409365:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757161.409369:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757161.409373:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787757163.713719:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757163.713734:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757163.713739:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757163.713744:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757163.713747:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757163.713761:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757163.713768:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757163.713775:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757163.713797:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659620480:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757163.713809:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757163.713818:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757163.713823:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757163.713829:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757163.713847:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.713884:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757163.713893:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133072896 : -123165576478720 : ffff8ffb4790c000) 00000100:00000001:2.0:1787757163.713906:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757163.713984:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757163.714031:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757163.714043:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757163.714051:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.714057:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757163.714076:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.714089:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304891585 00000100:00000001:2.0:1787757163.714107:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757163.714119:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304891585 00000100:00000001:2.0:1787757163.714144:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757163.714156:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757163.714164:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.714173:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757163.714251:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757163/real 1787757163] req@ffff8ffb4790c000 x1874588659620480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757163.714282:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757163.714293:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757163.714331:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757163.714342:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757163.714355:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714368:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757163.714379:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757163.714417:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c000 x1874588659620480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757163.714472:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714491:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757163.714505:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714513:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757163.714518:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714524:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757163.714530:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757163.714545:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757163.714557:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757163.714568:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757163.714585:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659620480:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757163.714615:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714625:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757163.714632:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757163.714647:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757163.714659:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757163.714666:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757163.714670:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757163.714676:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757163.714680:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757163.714684:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757163.714686:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714690:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714701:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757163.714705:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.714713:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757163.714716:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757163.714718:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757163.714721:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757163.714723:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757163.714727:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757163.714730:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757163.714733:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757163.714739:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659620608:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757163.714744:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757163.714747:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757163.714751:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757163.714754:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757163.714760:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.714773:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757163.714775:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133080064 : -123165576471552 : ffff8ffb4790dc00) 00000100:00000001:2.0:1787757163.714782:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757163.714811:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757163.714828:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757163.714835:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757163.714837:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.714840:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757163.714847:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.714852:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304890894 00000100:00000001:2.0:1787757163.714856:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304891585 00000100:00000001:2.0:1787757163.714859:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304890894 00000100:00000001:2.0:1787757163.714864:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304891585 00000100:00000001:2.0:1787757163.714869:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757163.714873:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.714876:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757163.714890:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757163/real 1787757163] req@ffff8ffb4790dc00 x1874588659620608/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757163.714911:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757163.714915:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757163.714921:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757163.714925:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757163.714931:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714936:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757163.714939:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757163.714952:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790dc00 x1874588659620608/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757163.714972:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714979:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757163.714984:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714987:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757163.714990:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.714993:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757163.715039:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757163.715045:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757163.715054:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757163.715057:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757163.715066:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659620608:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757163.715073:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757163.715075:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757163.715077:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757163.715081:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757163.715085:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757163.715088:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757163.715093:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659620736:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757163.715097:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757163.715101:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757163.715103:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757163.715105:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757163.715109:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715119:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757163.715121:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:2.0:1787757163.715129:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757163.715152:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757163.715160:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757163.715164:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757163.715166:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715169:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757163.715173:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715180:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304890891 00000100:00000001:2.0:1787757163.715227:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304891585 00000100:00000001:2.0:1787757163.715232:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304890892 00000100:00000001:2.0:1787757163.715236:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304891585 00000100:00000001:2.0:1787757163.715240:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757163.715243:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715247:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757163.715255:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757163/real 1787757163] req@ffff8ffb4790f480 x1874588659620736/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757163.715267:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757163.715269:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757163.715274:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757163.715277:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757163.715281:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715285:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757163.715288:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757163.715295:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659620736/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757163.715343:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715350:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757163.715357:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715362:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757163.715365:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715369:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757163.715399:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757163.715406:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757163.715416:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757163.715421:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757163.715428:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659620736:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757163.715437:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757163.715441:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757163.715443:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757163.715447:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757163.715450:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757163.715453:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757163.715457:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659620864:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757163.715461:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757163.715465:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757163.715468:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757163.715471:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757163.715476:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715485:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757163.715487:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:2.0:1787757163.715493:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757163.715514:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757163.715521:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757163.715524:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757163.715526:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715529:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757163.715532:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715537:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304890892 00000100:00000001:2.0:1787757163.715542:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304891585 00000100:00000001:2.0:1787757163.715545:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304890892 00000100:00000001:2.0:1787757163.715547:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304891585 00000100:00000001:2.0:1787757163.715550:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757163.715552:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715555:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757163.715562:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757163/real 1787757163] req@ffff8ffb4790ca80 x1874588659620864/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757163.715572:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757163.715573:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757163.715577:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757163.715580:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757163.715585:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715588:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757163.715590:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757163.715595:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659620864/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757163.715605:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715608:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757163.715613:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715615:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757163.715617:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715619:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757163.715640:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757163.715644:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757163.715650:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757163.715654:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757163.715658:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659620864:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757163.715667:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715672:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757163.715680:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757163.715688:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757163.715697:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757163.715699:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757163.715702:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757163.715707:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757163.715710:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757163.715713:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757163.715715:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715719:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757163.715722:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757163.715724:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757163.715728:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757163.715731:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757163.715734:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757163.715740:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757163.715743:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757163.715748:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757163.715787:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715792:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757163.715795:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757163.715800:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757163.715807:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757163.715810:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757163.715815:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757163.715825:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757163.715829:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757163.715837:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757163.715840:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715844:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757163.715853:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757163.715856:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757163.715861:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757163.715865:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757164.481496:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757164.481502:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757164.481506:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757165.505479:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757165.505485:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757165.505490:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757168.577425:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757168.577431:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757168.577435:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757168.833306:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757168.833315:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757168.833317:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757168.833320:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757168.833324:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757168.833330:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10223 00000100:00080000:2.0:1787757168.833339:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757168.833342:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757168.833343:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757168.833347:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757168.833349:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757168.833351:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757168.833353:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757168.833354:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757168.833355:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757168.833358:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757168.833360:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757168.833362:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757168.833365:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757168.833367:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833369:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757168.833370:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757168.833381:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757168.833382:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833385:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757168.833387:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757168.833391:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833401:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757168.833407:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659620992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757168.833416:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757168.833419:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833433:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757168.833435:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757168.833438:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757168.833439:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757168.833440:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757168.833442:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757168.833444:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10223 00000100:00080000:2.0:1787757168.833446:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757168.833447:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757168.833449:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757168.833450:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757168.833451:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757168.833453:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757168.833454:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757168.833455:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757168.833457:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757168.833460:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757168.833462:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757168.833463:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757168.833467:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757168.833468:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833470:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757168.833475:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757168.833480:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833483:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757168.833488:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757168.833491:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833496:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757168.833497:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757168.833499:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833505:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757168.833511:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659621120/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757168.833527:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757168.833529:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833534:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757168.833538:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757168.833541:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757168.833543:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757168.833546:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757168.833549:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757168.833552:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10223 00000100:00080000:2.0:1787757168.833554:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757168.833555:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757168.833556:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757168.833558:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757168.833559:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757168.833561:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757168.833565:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757168.833568:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757168.833570:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757168.833574:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757168.833575:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757168.833576:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757168.833578:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757168.833580:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833582:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757168.833584:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757168.833585:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833586:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757168.833590:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757168.833591:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833593:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757168.833593:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757168.833594:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833598:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757168.833601:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659621248/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757168.833612:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757168.833614:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833616:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757168.833618:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757168.833621:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757168.833624:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757168.833626:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757168.833628:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757168.833633:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10223 00000100:00080000:2.0:1787757168.833639:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757168.833642:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757168.833643:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757168.833649:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757168.833650:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757168.833652:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757168.833657:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757168.833658:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757168.833659:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757168.833661:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757168.833664:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757168.833669:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757168.833671:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757168.833673:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833677:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757168.833678:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757168.833680:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833684:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757168.833687:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757168.833688:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833689:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757168.833689:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757168.833690:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833693:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757168.833700:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659621376/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757168.833708:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757168.833712:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757168.833716:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757172.673386:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757172.673390:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757172.673393:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757176.769498:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757176.769504:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757176.769508:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757179.841473:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757179.841483:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757179.841492:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757180.865565:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757180.865572:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757180.865577:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757183.937435:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757183.937443:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757183.937449:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757184.961353:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757184.961357:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757184.961361:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757188.033471:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757188.033479:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757188.033483:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757189.057419:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757189.057425:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757189.057439:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757189.313412:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757189.313425:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757189.313429:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757189.313433:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757189.313439:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757189.313448:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10244 00000100:00080000:3.0:1787757189.313461:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757189.313464:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757189.313467:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757189.313472:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757189.313475:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757189.313493:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757189.313496:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757189.313499:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757189.313501:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757189.313507:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757189.313510:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757189.313514:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757189.313518:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757189.313522:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313524:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757189.313527:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757189.313540:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757189.313543:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313547:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757189.313550:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757189.313555:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313579:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757189.313592:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659623040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757189.313607:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757189.313612:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313632:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757189.313637:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757189.313641:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757189.313643:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757189.313646:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757189.313650:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757189.313652:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10244 00000100:00080000:3.0:1787757189.313657:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757189.313659:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757189.313661:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757189.313663:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757189.313665:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757189.313668:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757189.313671:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757189.313674:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757189.313675:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757189.313679:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757189.313681:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757189.313683:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757189.313695:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757189.313697:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313700:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757189.313706:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757189.313710:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313712:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757189.313717:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757189.313718:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313721:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757189.313723:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757189.313725:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313731:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757189.313738:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659623168/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757189.313749:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757189.313752:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313756:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757189.313760:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757189.313765:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757189.313767:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757189.313769:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757189.313774:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757189.313778:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10244 00000100:00080000:3.0:1787757189.313793:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757189.313797:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757189.313801:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757189.313813:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757189.313822:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757189.313827:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757189.313835:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757189.313840:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757189.313845:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757189.313887:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757189.313892:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757189.313897:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757189.313903:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757189.313920:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313928:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757189.313932:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757189.313944:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313950:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757189.313959:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757189.313962:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313967:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757189.313971:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757189.313974:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.313983:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757189.313993:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659623296/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757189.314022:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757189.314040:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.314049:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757189.314057:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757189.314064:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757189.314067:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757189.314071:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757189.314081:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757189.314088:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10244 00000100:00080000:3.0:1787757189.314097:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757189.314106:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757189.314110:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757189.314117:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757189.314119:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757189.314138:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757189.314149:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757189.314236:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757189.314244:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757189.314265:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757189.314281:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757189.314305:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757189.314317:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757189.314328:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.314336:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757189.314343:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757189.314354:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.314367:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757189.314383:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757189.314392:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.314408:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757189.314418:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757189.314436:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.314457:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757189.314474:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659623424/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757189.314521:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757189.314528:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757189.314537:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757190.081431:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757190.081436:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757190.081441:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757193.153424:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757193.153432:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757193.153438:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757194.177524:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757194.177530:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757194.177534:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787757194.433833:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757194.433841:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757194.433847:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757194.433851:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757194.433856:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757194.433864:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757194.433869:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757194.433876:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757194.433886:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659623552:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757194.433895:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757194.433902:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757194.433907:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757194.433912:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757194.433921:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.433948:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757194.433951:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:2.0:1787757194.433959:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757194.434046:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434054:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304922305 00000100:00000001:2.0:1787757194.434071:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757194.434088:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304922305 00000100:00000001:2.0:1787757194.434096:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757194.434106:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757194.434112:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434119:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434124:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757194.434136:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757194.434141:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787757194.434146:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757194.434154:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757194.434157:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757194.434160:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757194.434163:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757194.434167:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757194.434171:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757194.434175:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757194.434183:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659623680:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757194.434253:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757194.434258:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757194.434261:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757194.434265:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757194.434273:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434287:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757194.434289:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:2.0:1787757194.434298:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757194.434323:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434331:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304921614 00000100:00000001:2.0:1787757194.434336:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304922305 00000100:00000001:2.0:1787757194.434340:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304921614 00000100:00000001:2.0:1787757194.434345:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304922305 00000100:00000001:2.0:1787757194.434350:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757194.434355:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434361:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434366:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757194.434373:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757194.434377:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787757194.434382:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757194.434385:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757194.434388:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757194.434390:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757194.434393:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757194.434397:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757194.434401:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757194.434405:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757194.434411:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659623808:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757194.434417:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757194.434420:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757194.434422:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757194.434426:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757194.434430:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434440:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757194.434443:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:2.0:1787757194.434449:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757194.434469:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434475:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304921611 00000100:00000001:2.0:1787757194.434480:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304922305 00000100:00000001:2.0:1787757194.434486:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304921611 00000100:00000001:2.0:1787757194.434490:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304922305 00000100:00000001:2.0:1787757194.434493:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757194.434496:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434499:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434502:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757194.434505:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757194.434507:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787757194.434510:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757194.434511:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757194.434513:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757194.434515:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757194.434517:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757194.434520:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757194.434522:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757194.434524:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757194.434528:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659623936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757194.434532:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757194.434535:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757194.434537:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757194.434539:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757194.434543:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434549:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757194.434551:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:2.0:1787757194.434556:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757194.434567:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434571:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304921611 00000100:00000001:2.0:1787757194.434574:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304922305 00000100:00000001:2.0:1787757194.434576:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304921611 00000100:00000001:2.0:1787757194.434579:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304922305 00000100:00000001:2.0:1787757194.434581:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757194.434583:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434587:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434589:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757194.434591:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757194.434593:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787757194.434595:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757194.434597:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757194.434599:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757194.434601:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757197.249550:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757197.249557:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757197.249562:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787757197.505622:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787757197.505742:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787757197.505774:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757197.505803:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757197.505851:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757197.505857:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757197.505862:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757197.505883:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757197.505894:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757197.505900:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757197.505902:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757197.505906:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757197.505915:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757197.505924:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757197.505928:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757197.505931:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757197.505934:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757197.505943:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757197.505952:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757197.505956:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757197.505959:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757197.505962:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757197.506125:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757197.506140:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757197.506146:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757197.506170:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757194/real 1787757197] req@ffff8ffb4790e680 x1874588659623936/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757197.506275:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757197.506280:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757197.506296:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757197.506303:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757197.506307:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506313:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757197.506317:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757197.506328:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659623936/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757197.506344:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506351:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757197.506357:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506361:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757197.506363:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506368:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757197.506394:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757197.506402:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757197.506412:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757197.506417:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757197.506432:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659623936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757197.506447:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757197.506458:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757194/real 1787757197] req@ffff8ffb4790d180 x1874588659623808/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757197.506478:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757197.506483:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757197.506494:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757197.506501:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757197.506504:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506508:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757197.506511:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757197.506520:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659623808/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757197.506534:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506538:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757197.506542:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506545:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757197.506547:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506550:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757197.506573:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757197.506579:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757197.506585:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757197.506589:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757197.506595:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659623808:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757197.506602:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757197.506611:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757194/real 1787757197] req@ffff8ffb4790fb80 x1874588659623680/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757197.506627:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757197.506629:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757197.506636:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757197.506639:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757197.506642:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506645:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757197.506648:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757197.506655:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659623680/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757197.506668:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506672:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757197.506677:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506680:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757197.506682:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506685:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757197.506744:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757197.506752:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757197.506762:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757197.506769:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757197.506778:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659623680:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757197.506789:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757197.506798:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757194/real 1787757197] req@ffff8ffb4790ed80 x1874588659623552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757197.506828:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757197.506830:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757197.506838:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757197.506841:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757197.506844:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506848:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757197.506851:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757197.506860:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659623552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757197.506875:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506881:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757197.506886:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506889:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757197.506892:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506895:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757197.506898:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757197.506903:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757197.506910:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757197.506914:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757197.506920:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659623552:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757197.506928:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506934:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757197.506938:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757197.506943:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757197.506953:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757197.506958:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757197.506963:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757197.506971:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757197.506974:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757197.506980:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757197.506983:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.506987:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757197.506989:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757197.506991:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757197.506995:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757197.506997:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757197.506999:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757197.507004:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757197.507006:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757197.507009:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757197.507011:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.507015:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757197.507017:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757197.507019:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757197.507025:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757197.507027:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757197.507029:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757197.507033:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757197.507034:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757197.507038:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757197.507040:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.507043:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757197.507045:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757197.507048:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757197.507051:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757197.507053:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757197.507055:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757197.507057:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757197.507060:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757197.507063:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757197.507065:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.507069:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757197.507081:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757197.507084:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757197.507090:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757197.507093:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757198.273444:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757198.273450:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757198.273456:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757201.345448:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757201.345458:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757201.345464:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757202.369401:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757202.369406:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757202.369411:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757204.673361:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757204.673374:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757204.673378:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757204.673382:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757204.673388:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757204.673397:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10259 00000100:00080000:2.0:1787757204.673412:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757204.673416:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757204.673419:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757204.673424:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757204.673427:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757204.673431:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757204.673434:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757204.673437:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757204.673439:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757204.673445:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757204.673451:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757204.673455:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757204.673459:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757204.673463:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673466:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757204.673469:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757204.673484:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757204.673486:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673490:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757204.673494:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757204.673499:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673513:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757204.673526:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659624576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757204.673545:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757204.673553:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673627:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757204.673635:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757204.673643:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757204.673646:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757204.673650:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757204.673657:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757204.673663:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10259 00000100:00080000:2.0:1787757204.673672:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757204.673676:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757204.673679:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757204.673682:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757204.673685:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757204.673688:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757204.673691:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757204.673696:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757204.673700:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757204.673713:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757204.673728:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757204.673742:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757204.673760:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757204.673767:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673778:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757204.673792:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757204.673813:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673829:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757204.673847:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757204.673856:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673867:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757204.673873:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757204.673880:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673899:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757204.673913:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659624704/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757204.673933:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757204.673940:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.673949:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757204.673957:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757204.673962:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757204.673964:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757204.673967:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757204.673971:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757204.673976:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10259 00000100:00080000:2.0:1787757204.673982:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757204.673985:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757204.673987:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757204.673991:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757204.673992:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757204.673996:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757204.673999:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757204.674003:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757204.674006:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757204.674016:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757204.674021:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757204.674025:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757204.674033:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757204.674038:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674041:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757204.674047:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757204.674062:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674069:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757204.674081:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757204.674087:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674093:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757204.674102:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757204.674108:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674121:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757204.674138:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659624832/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757204.674160:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757204.674167:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674277:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757204.674291:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757204.674308:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757204.674323:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757204.674329:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757204.674342:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757204.674358:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10259 00000100:00080000:2.0:1787757204.674380:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757204.674391:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757204.674397:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757204.674409:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757204.674413:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757204.674424:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757204.674434:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757204.674445:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757204.674450:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757204.674463:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757204.674469:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757204.674484:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757204.674496:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757204.674510:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674516:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757204.674530:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757204.674550:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674567:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757204.674620:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757204.674629:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674638:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757204.674643:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757204.674654:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674668:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757204.674689:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659624960/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757204.674727:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757204.674743:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757204.674751:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757205.441512:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757205.441520:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757205.441526:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757206.465518:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757206.465527:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757206.465534:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757208.513431:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757208.513437:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757208.513456:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757209.537475:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757209.537483:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757209.537490:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757209.793348:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757209.793364:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757209.793369:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757209.793374:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757209.793380:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757209.793391:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10264 00000100:00080000:3.0:1787757209.793406:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757209.793410:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757209.793414:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757209.793420:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757209.793423:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757209.793427:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757209.793430:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757209.793434:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757209.793436:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757209.793444:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757209.793448:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757209.793452:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757209.793457:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757209.793462:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793465:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757209.793468:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757209.793520:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757209.793524:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793530:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757209.793534:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757209.793540:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793556:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757209.793569:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659625088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757209.793584:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757209.793592:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793625:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757209.793633:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757209.793641:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757209.793646:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757209.793651:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757209.793659:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757209.793666:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10264 00000100:00080000:3.0:1787757209.793676:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757209.793688:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757209.793696:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757209.793709:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757209.793717:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757209.793724:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757209.793731:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757209.793741:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757209.793749:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757209.793776:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757209.793784:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757209.793789:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757209.793800:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757209.793810:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793815:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757209.793824:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757209.793841:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793847:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757209.793855:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757209.793857:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793859:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757209.793862:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757209.793877:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793903:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757209.793919:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659625216/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757209.793943:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757209.793948:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.793955:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757209.793959:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757209.793965:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757209.793969:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757209.793975:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757209.793986:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757209.793996:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10264 00000100:00080000:3.0:1787757209.794009:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757209.794018:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757209.794027:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757209.794041:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757209.794054:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757209.794065:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757209.794077:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757209.794087:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757209.794094:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757209.794105:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757209.794116:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757209.794121:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757209.794130:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757209.794137:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794141:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757209.794147:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757209.794156:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794163:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757209.794175:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757209.794232:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794250:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757209.794258:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757209.794269:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794288:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757209.794306:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659625344/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757209.794346:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757209.794362:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794377:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757209.794393:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757209.794413:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757209.794418:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757209.794429:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757209.794457:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757209.794478:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10264 00000100:00080000:3.0:1787757209.794519:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757209.794527:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757209.794531:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757209.794539:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757209.794542:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757209.794547:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757209.794559:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757209.794562:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757209.794572:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757209.794595:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757209.794613:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757209.794623:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757209.794636:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757209.794654:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794667:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757209.794676:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757209.794695:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794704:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757209.794721:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757209.794730:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794736:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757209.794738:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757209.794743:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794755:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757209.794773:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659625472/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757209.794808:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757209.794819:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757209.794831:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757210.561409:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757210.561415:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757210.561419:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757212.609479:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757212.609492:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757212.609501:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757213.633504:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757213.633510:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757213.633515:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757214.657459:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757214.657468:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757214.657475:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757214.913501:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757214.913514:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757214.913518:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757214.913522:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757214.913528:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757214.913538:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10269 00000100:00080000:2.0:1787757214.913553:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757214.913557:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757214.913560:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757214.913565:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757214.913569:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757214.913573:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757214.913576:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757214.913579:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757214.913581:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757214.913587:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757214.913590:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757214.913593:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757214.913597:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757214.913601:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913604:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757214.913607:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757214.913621:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757214.913624:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913628:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757214.913631:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757214.913636:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913651:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757214.913663:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659625600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757214.913677:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757214.913684:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913711:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757214.913715:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757214.913720:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757214.913722:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757214.913724:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757214.913728:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757214.913731:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10269 00000100:00080000:2.0:1787757214.913735:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757214.913737:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757214.913739:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757214.913741:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757214.913743:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757214.913746:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757214.913749:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757214.913752:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757214.913753:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757214.913757:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757214.913761:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757214.913764:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757214.913771:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757214.913776:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913778:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757214.913785:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757214.913791:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913794:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757214.913798:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757214.913800:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913803:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757214.913806:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757214.913811:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913822:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757214.913837:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659625728/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757214.913855:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757214.913863:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.913876:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757214.913884:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757214.913895:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757214.913906:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757214.913911:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757214.913922:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757214.913934:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10269 00000100:00080000:2.0:1787757214.913954:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757214.913976:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757214.913982:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757214.913993:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757214.914001:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757214.914006:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757214.914012:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757214.914015:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757214.914017:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757214.914025:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757214.914028:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757214.914031:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757214.914038:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757214.914045:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914051:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757214.914057:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757214.914066:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914069:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757214.914077:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757214.914080:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914084:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757214.914087:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757214.914090:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914098:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757214.914108:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659625856/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757214.914119:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757214.914123:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914130:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757214.914142:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757214.914149:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757214.914153:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757214.914157:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757214.914162:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757214.914172:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10269 00000100:00080000:2.0:1787757214.914244:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757214.914251:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757214.914256:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757214.914264:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757214.914267:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757214.914282:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757214.914290:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757214.914294:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757214.914296:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757214.914308:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757214.914312:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757214.914316:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757214.914324:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757214.914341:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914348:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757214.914359:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757214.914368:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914378:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757214.914385:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757214.914386:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914388:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757214.914390:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757214.914394:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914439:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757214.914462:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659625984/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757214.914483:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757214.914490:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757214.914500:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757216.705491:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757216.705497:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757216.705502:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757217.729515:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757217.729521:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757217.729526:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757218.753504:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757218.753511:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757218.753516:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787757220.033672:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757220.033684:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757220.033691:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757220.033697:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757220.033701:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757220.033711:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757220.033720:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757220.033730:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757220.033744:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659626112:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757220.033754:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757220.033763:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757220.033769:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757220.033777:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757220.033785:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.033814:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757220.033820:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:2.0:1787757220.033830:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757220.033894:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757220.033918:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757220.033928:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757220.033935:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.033942:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757220.033954:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.033969:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4304947905 00000100:00000001:2.0:1787757220.033988:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757220.034003:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4304947905 00000100:00000001:2.0:1787757220.034013:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757220.034021:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757220.034028:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034036:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757220.034049:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757220/real 1787757220] req@ffff8ffb4790d880 x1874588659626112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757220.034075:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757220.034081:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757220.034088:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757220.034096:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757220.034103:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034110:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757220.034118:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757220.034134:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659626112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757220.034155:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034163:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757220.034170:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034177:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757220.034239:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034249:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757220.034255:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757220.034266:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757220.034280:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757220.034290:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757220.034334:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659626112:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757220.034349:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034353:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757220.034356:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757220.034360:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757220.034366:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757220.034370:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757220.034374:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757220.034380:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757220.034383:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757220.034388:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757220.034390:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034395:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034406:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757220.034409:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034412:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757220.034414:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757220.034416:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757220.034418:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757220.034420:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757220.034423:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757220.034426:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757220.034429:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757220.034440:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659626240:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757220.034451:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757220.034458:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757220.034465:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757220.034471:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757220.034480:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034496:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757220.034504:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:2.0:1787757220.034517:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757220.034550:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757220.034560:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757220.034567:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757220.034572:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034575:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757220.034580:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034591:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304947214 00000100:00000001:2.0:1787757220.034600:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304947905 00000100:00000001:2.0:1787757220.034609:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304947214 00000100:00000001:2.0:1787757220.034617:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304947905 00000100:00000001:2.0:1787757220.034620:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757220.034631:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034633:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757220.034639:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757220/real 1787757220] req@ffff8ffb4790f100 x1874588659626240/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757220.034659:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757220.034665:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757220.034669:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757220.034675:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757220.034680:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034688:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757220.034693:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757220.034708:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659626240/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757220.034732:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034737:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757220.034741:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034743:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757220.034744:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034747:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757220.034768:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757220.034772:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757220.034776:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757220.034780:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757220.034785:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659626240:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757220.034790:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757220.034792:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757220.034794:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757220.034797:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757220.034799:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757220.034801:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757220.034805:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659626368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757220.034809:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757220.034811:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757220.034813:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757220.034815:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757220.034819:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034825:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757220.034826:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:2.0:1787757220.034831:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757220.034844:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757220.034849:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757220.034852:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757220.034854:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034856:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757220.034859:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034863:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304947211 00000100:00000001:2.0:1787757220.034867:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304947905 00000100:00000001:2.0:1787757220.034869:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304947211 00000100:00000001:2.0:1787757220.034872:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304947905 00000100:00000001:2.0:1787757220.034875:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757220.034878:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.034880:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757220.034885:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757220/real 1787757220] req@ffff8ffb4790c380 x1874588659626368/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757220.034903:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757220.034904:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757220.034907:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757220.034909:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757220.034911:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034913:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757220.034914:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757220.034919:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659626368/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757220.034927:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034930:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757220.034933:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034935:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757220.034937:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034939:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757220.034952:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757220.034955:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757220.034958:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757220.034961:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757220.034965:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659626368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757220.034969:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034972:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757220.034974:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757220.034975:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757220.034978:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757220.034980:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757220.034981:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757220.034984:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757220.034986:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757220.034988:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757220.034990:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.034993:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757220.034994:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757220.034996:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757220.034999:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757220.035001:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757220.035002:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757220.035005:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757220.035006:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757220.035008:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757220.035010:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035012:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035016:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757220.035017:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.035020:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757220.035021:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757220.035023:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757220.035024:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757220.035026:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757220.035028:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757220.035030:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757220.035033:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757220.035037:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659626496:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757220.035041:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757220.035043:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757220.035044:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757220.035046:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757220.035049:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.035054:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757220.035056:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:2.0:1787757220.035060:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757220.035073:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757220.035078:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757220.035080:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757220.035082:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.035084:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757220.035087:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.035091:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4304947211 00000100:00000001:2.0:1787757220.035094:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4304947905 00000100:00000001:2.0:1787757220.035097:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4304947211 00000100:00000001:2.0:1787757220.035100:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4304947905 00000100:00000001:2.0:1787757220.035103:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757220.035106:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.035108:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757220.035113:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757220/real 1787757220] req@ffff8ffb4790ce00 x1874588659626496/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757220.035121:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757220.035123:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757220.035126:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757220.035127:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757220.035130:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035132:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757220.035134:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757220.035138:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659626496/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757220.035146:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035148:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757220.035151:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035153:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757220.035154:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035156:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757220.035170:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757220.035173:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757220.035177:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757220.035180:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757220.035212:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659626496:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757220.035217:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035220:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757220.035222:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757220.035224:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757220.035227:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757220.035229:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757220.035230:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757220.035233:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757220.035234:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757220.035237:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757220.035239:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035241:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757220.035245:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757220.035247:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757220.035249:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757220.035251:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757220.801467:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757220.801472:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757220.801477:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757221.825436:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757221.825442:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757221.825446:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757222.849475:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757222.849481:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757222.849485:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757224.897444:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757224.897451:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757224.897456:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757225.153385:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757225.153397:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757225.153400:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757225.153404:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757225.153410:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757225.153419:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10279 00000100:00080000:2.0:1787757225.153432:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757225.153437:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757225.153439:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757225.153444:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757225.153447:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757225.153451:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757225.153455:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757225.153457:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757225.153459:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757225.153465:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757225.153469:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757225.153488:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757225.153492:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757225.153496:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153499:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757225.153503:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757225.153518:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757225.153521:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153526:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757225.153531:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757225.153539:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153558:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757225.153575:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659626624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757225.153596:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757225.153605:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153639:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757225.153646:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757225.153653:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757225.153656:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757225.153659:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757225.153663:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757225.153667:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10279 00000100:00080000:2.0:1787757225.153673:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757225.153676:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757225.153678:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757225.153682:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757225.153683:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757225.153687:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757225.153689:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757225.153692:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757225.153695:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757225.153699:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757225.153702:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757225.153705:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757225.153712:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757225.153714:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153717:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757225.153723:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757225.153728:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153731:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757225.153738:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757225.153741:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153743:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757225.153745:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757225.153749:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153758:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757225.153768:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659626752/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757225.153792:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757225.153800:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153807:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757225.153813:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757225.153824:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757225.153846:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757225.153854:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757225.153860:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757225.153870:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10279 00000100:00080000:2.0:1787757225.153877:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757225.153882:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757225.153885:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757225.153893:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757225.153896:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757225.153899:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757225.153902:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757225.153905:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757225.153907:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757225.153913:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757225.153922:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757225.153934:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757225.153940:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757225.153946:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153948:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757225.153951:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757225.153955:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153958:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757225.153965:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757225.153967:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153969:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757225.153972:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757225.153975:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.153983:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757225.153995:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659626880/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757225.154011:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757225.154014:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.154018:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757225.154025:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757225.154031:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757225.154036:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757225.154041:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757225.154048:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757225.154055:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10279 00000100:00080000:2.0:1787757225.154061:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757225.154067:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757225.154073:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757225.154079:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757225.154080:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757225.154083:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757225.154087:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757225.154096:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757225.154098:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757225.154105:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757225.154112:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757225.154116:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757225.154123:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757225.154125:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.154127:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757225.154129:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757225.154132:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.154134:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757225.154140:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757225.154146:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.154148:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757225.154149:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757225.154151:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.154155:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757225.154163:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659627008/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757225.154274:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757225.154281:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757225.154289:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757225.921366:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757225.921374:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757225.921380:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757226.945484:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757226.945491:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757226.945496:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757228.993396:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757228.993403:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757228.993407:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757230.017336:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757230.017340:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757230.017343:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757230.273255:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757230.273264:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757230.273266:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757230.273269:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757230.273273:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757230.273279:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10285 00000100:00080000:3.0:1787757230.273288:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757230.273290:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757230.273292:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757230.273296:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757230.273298:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757230.273301:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757230.273303:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757230.273304:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757230.273305:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757230.273308:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757230.273310:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757230.273312:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757230.273314:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757230.273316:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273318:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757230.273319:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757230.273328:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757230.273329:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273331:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757230.273333:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757230.273338:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273348:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757230.273354:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659627136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757230.273362:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757230.273366:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273378:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757230.273381:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757230.273385:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757230.273386:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757230.273388:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757230.273391:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757230.273394:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10285 00000100:00080000:3.0:1787757230.273397:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757230.273400:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757230.273401:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757230.273404:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757230.273405:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757230.273407:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757230.273410:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757230.273412:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757230.273413:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757230.273417:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757230.273419:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757230.273421:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757230.273426:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757230.273428:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273430:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757230.273435:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757230.273438:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273440:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757230.273444:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757230.273446:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273448:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757230.273450:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757230.273451:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273455:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757230.273461:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659627264/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757230.273468:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757230.273470:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273473:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757230.273475:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757230.273478:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757230.273480:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757230.273482:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757230.273485:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757230.273487:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10285 00000100:00080000:3.0:1787757230.273490:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757230.273492:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757230.273493:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757230.273495:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757230.273497:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757230.273498:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757230.273500:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757230.273502:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757230.273503:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757230.273506:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757230.273508:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757230.273509:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757230.273511:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757230.273514:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273515:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757230.273519:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757230.273521:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273523:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757230.273526:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757230.273527:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273529:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757230.273530:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757230.273532:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273536:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757230.273540:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659627392/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757230.273548:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757230.273549:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273551:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757230.273554:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757230.273558:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757230.273559:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757230.273560:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757230.273563:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757230.273566:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10285 00000100:00080000:3.0:1787757230.273569:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757230.273571:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757230.273573:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757230.273574:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757230.273575:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757230.273577:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757230.273579:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757230.273581:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757230.273583:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757230.273586:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757230.273589:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757230.273591:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757230.273593:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757230.273595:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273597:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757230.273600:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757230.273602:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273604:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757230.273608:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757230.273609:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273611:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757230.273613:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757230.273614:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273618:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757230.273623:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659627520/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757230.273630:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757230.273632:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757230.273635:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757231.041451:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757231.041457:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757231.041462:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757233.089330:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757233.089333:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757233.089337:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757234.113465:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757234.113471:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757234.113475:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757235.137431:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757235.137437:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757235.137442:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757235.393293:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757235.393326:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757235.393331:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757235.393337:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757235.393344:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757235.393367:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10290 00000100:00080000:3.0:1787757235.393385:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757235.393390:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757235.393394:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757235.393402:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757235.393407:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757235.393413:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757235.393418:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757235.393422:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757235.393425:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757235.393435:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757235.393441:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757235.393445:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757235.393452:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757235.393458:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.393462:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757235.393466:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757235.393490:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757235.393494:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.393498:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757235.393503:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757235.393510:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.393528:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757235.393545:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659627648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757235.393575:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757235.393584:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.393612:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757235.393620:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757235.393632:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757235.393641:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757235.393649:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757235.393665:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757235.393679:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10290 00000100:00080000:3.0:1787757235.393694:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757235.393702:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757235.394078:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757235.394095:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757235.394098:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757235.394102:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757235.394107:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757235.394116:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757235.394122:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757235.394140:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757235.394150:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757235.394159:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757235.394172:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757235.394182:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394306:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757235.394315:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757235.394323:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394328:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757235.394338:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757235.394341:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394344:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757235.394346:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757235.394350:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394366:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757235.394376:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659627776/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757235.394388:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757235.394391:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394408:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757235.394412:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757235.394416:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757235.394419:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757235.394422:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757235.394426:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757235.394429:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10290 00000100:00080000:3.0:1787757235.394436:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757235.394439:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757235.394441:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757235.394445:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757235.394447:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757235.394450:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757235.394454:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757235.394457:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757235.394459:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757235.394464:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757235.394467:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757235.394470:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757235.394489:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757235.394496:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394503:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757235.394507:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757235.394514:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394520:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757235.394528:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757235.394534:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394544:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757235.394547:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757235.394551:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394561:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757235.394575:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659627904/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757235.394598:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757235.394606:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394613:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757235.394617:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757235.394625:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757235.394630:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757235.394635:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757235.394641:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757235.394645:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10290 00000100:00080000:3.0:1787757235.394653:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757235.394656:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757235.394660:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757235.394666:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757235.394671:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757235.394674:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757235.394678:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757235.394685:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757235.394688:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757235.394699:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757235.394708:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757235.394713:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757235.394721:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757235.394726:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394735:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757235.394739:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757235.394748:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394757:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757235.394770:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757235.394774:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394780:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757235.394785:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757235.394788:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394795:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757235.394810:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659628032/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757235.394833:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757235.394841:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757235.394849:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757237.185487:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757237.185496:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757237.185503:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757238.209351:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757238.209355:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757238.209358:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757239.233454:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757239.233465:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757239.233474:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757240.513308:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757240.513322:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757240.513326:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757240.513330:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757240.513337:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757240.513346:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10295 00000100:00080000:3.0:1787757240.513359:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757240.513363:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757240.513365:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757240.513371:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757240.513375:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757240.513379:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757240.513382:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757240.513385:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757240.513387:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757240.513393:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757240.513396:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757240.513399:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757240.513404:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757240.513408:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513410:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757240.513414:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757240.513443:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757240.513446:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513450:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757240.513454:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757240.513459:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513474:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757240.513487:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659628160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757240.513501:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757240.513507:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513527:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757240.513531:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757240.513536:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757240.513538:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757240.513540:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757240.513543:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757240.513546:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10295 00000100:00080000:3.0:1787757240.513551:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757240.513553:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757240.513555:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757240.513558:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757240.513560:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757240.513563:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757240.513566:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757240.513569:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757240.513571:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757240.513575:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757240.513579:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757240.513582:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757240.513588:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757240.513593:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513596:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757240.513603:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757240.513608:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513612:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757240.513618:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757240.513624:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513634:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757240.513639:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757240.513644:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513659:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757240.513680:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659628288/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757240.513715:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757240.513728:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513739:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757240.513748:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757240.513755:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757240.513760:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757240.513770:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757240.513782:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757240.513789:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10295 00000100:00080000:3.0:1787757240.513797:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757240.513800:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757240.513802:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757240.513806:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757240.513809:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757240.513812:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757240.513815:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757240.513818:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757240.513820:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757240.513833:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757240.513841:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757240.513844:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757240.513850:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757240.513858:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513861:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757240.513870:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757240.513879:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513885:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757240.513892:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757240.513893:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513896:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757240.513897:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757240.513900:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513909:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757240.513926:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659628416/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757240.513940:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757240.513973:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.513979:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757240.513984:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757240.513993:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757240.513997:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757240.514003:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757240.514010:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757240.514018:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10295 00000100:00080000:3.0:1787757240.514027:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757240.514036:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757240.514040:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757240.514050:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757240.514055:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757240.514061:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757240.514071:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757240.514078:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757240.514087:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757240.514098:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757240.514107:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757240.514114:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757240.514124:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757240.514135:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.514143:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757240.514150:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757240.514162:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.514169:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757240.514232:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757240.514240:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.514252:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757240.514260:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757240.514267:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.514283:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757240.514305:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659628544/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757240.514336:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757240.514343:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757240.514350:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757241.281481:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757241.281490:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757241.281497:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757242.305490:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757242.305499:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757242.305505:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757243.329549:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757243.329556:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757243.329561:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757245.377478:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757245.377485:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757245.377490:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757245.633308:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757245.633321:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757245.633324:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757245.633329:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757245.633335:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757245.633357:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10300 00000100:00080000:3.0:1787757245.633370:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757245.633375:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757245.633377:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757245.633383:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757245.633386:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757245.633390:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757245.633394:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757245.633397:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757245.633398:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757245.633405:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757245.633408:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757245.633411:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757245.633415:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757245.633419:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633422:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757245.633424:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757245.633437:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757245.633439:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633443:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757245.633447:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757245.633453:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633466:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757245.633478:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659628672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757245.633492:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757245.633498:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633520:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757245.633525:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757245.633529:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757245.633531:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757245.633534:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757245.633538:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757245.633541:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10300 00000100:00080000:3.0:1787757245.633545:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757245.633548:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757245.633550:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757245.633553:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757245.633555:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757245.633558:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757245.633560:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757245.633562:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757245.633564:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757245.633568:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757245.633572:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757245.633574:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757245.633580:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757245.633584:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633586:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757245.633593:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757245.633597:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633600:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757245.633604:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757245.633606:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633608:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757245.633610:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757245.633613:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633637:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757245.633650:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659628800/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757245.633676:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757245.633683:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633692:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757245.633701:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757245.633731:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757245.633739:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757245.633743:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757245.633752:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757245.633763:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10300 00000100:00080000:3.0:1787757245.633770:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757245.633772:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757245.633773:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757245.633782:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757245.633790:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757245.633793:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757245.633797:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757245.633800:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757245.633801:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757245.633805:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757245.633811:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757245.633819:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757245.633826:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757245.633833:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633838:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757245.633841:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757245.633844:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633847:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757245.633854:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757245.633877:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633880:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757245.633882:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757245.633884:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633893:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757245.633900:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659628928/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757245.633918:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757245.633921:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.633925:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757245.633933:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757245.633941:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757245.633946:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757245.633949:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757245.633953:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757245.633959:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10300 00000100:00080000:3.0:1787757245.633964:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757245.633971:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757245.633974:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757245.633978:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757245.633985:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757245.633988:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757245.633998:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757245.634003:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757245.634008:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757245.634014:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757245.634019:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757245.634026:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757245.634034:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757245.634040:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.634043:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757245.634046:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757245.634055:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.634058:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757245.634065:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757245.634071:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.634078:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757245.634083:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757245.634090:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.634102:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757245.634115:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659629056/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757245.634222:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757245.634225:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757245.634230:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757246.401401:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757246.401409:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757246.401414:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757247.425458:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757247.425467:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757247.425474:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757249.473465:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757249.473471:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757249.473477:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757250.497504:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757250.497512:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757250.497517:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757251.521392:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757251.521399:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757251.521405:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757253.569562:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757253.569571:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757253.569577:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757254.593485:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757254.593491:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757254.593496:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757255.617537:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757255.617543:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757255.617547:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757257.665460:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757257.665487:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757257.665495:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757258.689423:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757258.689429:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757258.689434:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757259.713453:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757259.713461:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757259.713466:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757260.993357:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757260.993370:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757260.993376:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757260.993381:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757260.993388:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757260.993409:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10315 00000100:00080000:3.0:1787757260.993424:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757260.993428:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757260.993431:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757260.993436:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757260.993440:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757260.993444:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757260.993447:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757260.993450:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757260.993452:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757260.993458:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757260.993462:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757260.993465:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757260.993469:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757260.993473:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993476:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757260.993479:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757260.993492:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757260.993494:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993499:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757260.993502:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757260.993507:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993521:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757260.993533:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659630208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757260.993548:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757260.993554:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993573:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757260.993578:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757260.993650:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757260.993659:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757260.993668:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757260.993682:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757260.993695:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10315 00000100:00080000:3.0:1787757260.993712:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757260.993719:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757260.993725:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757260.993731:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757260.993736:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757260.993741:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757260.993748:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757260.993757:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757260.993764:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757260.993778:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757260.993782:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757260.993785:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757260.993793:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757260.993798:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993805:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757260.993815:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757260.993822:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993825:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757260.993834:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757260.993836:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993838:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757260.993840:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757260.993843:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993855:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757260.993862:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659630336/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757260.993915:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757260.993926:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.993942:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757260.993954:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757260.993973:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757260.993981:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757260.993990:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757260.993999:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757260.994009:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10315 00000100:00080000:3.0:1787757260.994022:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757260.994032:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757260.994038:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757260.994047:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757260.994050:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757260.994058:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757260.994062:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757260.994073:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757260.994077:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757260.994091:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757260.994099:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757260.994107:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757260.994118:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757260.994128:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994137:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757260.994147:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757260.994157:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994167:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757260.994180:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757260.994255:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994261:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757260.994264:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757260.994268:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994285:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757260.994297:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659630464/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757260.994309:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757260.994313:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994319:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757260.994331:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757260.994336:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757260.994342:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757260.994348:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757260.994360:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757260.994367:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10315 00000100:00080000:3.0:1787757260.994380:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757260.994387:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757260.994392:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757260.994397:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757260.994398:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757260.994401:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757260.994404:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757260.994407:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757260.994409:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757260.994416:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757260.994429:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757260.994443:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757260.994453:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757260.994462:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994471:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757260.994478:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757260.994487:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994491:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757260.994506:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757260.994514:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994522:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757260.994529:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757260.994539:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994559:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757260.994581:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659630592/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757260.994643:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757260.994652:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757260.994662:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757261.761515:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757261.761521:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757261.761526:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757262.785408:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757262.785414:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757262.785419:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787757264.065578:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787757264.065634:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787757264.065653:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757264.065668:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757264.065692:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757264.065695:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757264.065698:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757264.065710:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757264.065717:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757264.065721:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757264.065723:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757264.065726:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757264.065733:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757264.065740:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757264.065748:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757264.065753:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757264.065757:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757264.065767:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757264.065776:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757264.065789:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757264.065792:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757264.065797:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:2.0:1787757265.857360:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757265.857364:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757265.857367:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757266.881532:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757266.881542:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757266.881549:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757269.953417:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757269.953424:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757269.953450:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757270.977548:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757270.977554:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757270.977558:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757271.233315:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757271.233327:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757271.233331:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757271.233336:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757271.233342:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757271.233352:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10325 00000100:00080000:3.0:1787757271.233371:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757271.233375:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757271.233378:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757271.233385:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757271.233390:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757271.233394:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757271.233399:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757271.233403:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757271.233425:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757271.233435:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757271.233439:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757271.233443:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757271.233460:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757271.233465:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233468:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757271.233471:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757271.233484:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757271.233487:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233492:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757271.233496:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757271.233500:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233513:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757271.233527:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659631232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757271.233542:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757271.233548:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233568:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757271.233574:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757271.233579:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757271.233581:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757271.233584:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757271.233590:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757271.233595:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10325 00000100:00080000:3.0:1787757271.233603:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757271.233606:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757271.233609:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757271.233612:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757271.233615:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757271.233619:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757271.233627:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757271.233635:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757271.233645:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757271.233660:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757271.233672:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757271.233685:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757271.233698:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757271.233708:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233715:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757271.233727:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757271.233739:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233743:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757271.233751:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757271.233758:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233763:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757271.233879:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757271.233886:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233903:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757271.233923:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659631360/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757271.233940:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757271.233946:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.233960:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757271.233970:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757271.233975:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757271.233981:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757271.233986:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757271.233996:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757271.234004:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10325 00000100:00080000:3.0:1787757271.234012:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757271.234016:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757271.234018:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757271.234021:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757271.234023:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757271.234211:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757271.234218:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757271.234222:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757271.234224:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757271.234232:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757271.234238:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757271.234247:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757271.234255:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757271.234263:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234485:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757271.234491:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757271.234518:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234523:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757271.234534:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757271.234538:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234545:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757271.234552:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757271.234561:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234576:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757271.234598:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659631488/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757271.234644:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757271.234657:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234672:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757271.234683:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757271.234699:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757271.234708:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757271.234716:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757271.234731:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757271.234740:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10325 00000100:00080000:3.0:1787757271.234753:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757271.234762:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757271.234768:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757271.234778:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757271.234785:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757271.234792:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757271.234802:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757271.234814:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757271.234823:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757271.234836:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757271.234842:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757271.234847:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757271.234853:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757271.234865:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234868:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757271.234874:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757271.234880:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234887:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757271.234901:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757271.234906:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234910:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757271.234915:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757271.234921:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234931:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757271.234944:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659631616/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757271.234971:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757271.234983:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757271.234993:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757274.049540:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757274.049546:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757274.049551:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757275.073468:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757275.073477:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757275.073482:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757276.353372:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757276.353386:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757276.353389:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757276.353394:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757276.353400:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757276.353409:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10331 00000100:00080000:3.0:1787757276.353422:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757276.353425:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757276.353429:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757276.353435:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757276.353438:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757276.353442:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757276.353445:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757276.353448:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757276.353450:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757276.353456:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757276.353459:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757276.353463:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757276.353467:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757276.353471:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353473:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757276.353477:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757276.353490:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757276.353492:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353497:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757276.353500:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757276.353505:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353520:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757276.353531:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659631744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757276.353545:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757276.353551:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353575:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757276.353581:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757276.353588:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757276.353601:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757276.353604:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757276.353610:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757276.353616:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10331 00000100:00080000:3.0:1787757276.353624:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757276.353630:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757276.353633:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757276.353638:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757276.353640:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757276.353645:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757276.353649:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757276.353654:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757276.353657:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757276.353663:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757276.353668:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757276.353671:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757276.353680:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757276.353684:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353688:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757276.353697:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757276.353703:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353707:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757276.353715:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757276.353718:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353722:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757276.353725:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757276.353728:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353737:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757276.353747:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659631872/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757276.353762:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757276.353767:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353773:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757276.353778:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757276.353785:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757276.353788:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757276.353791:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757276.353797:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757276.353801:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10331 00000100:00080000:3.0:1787757276.353807:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757276.353811:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757276.353814:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757276.353831:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757276.353833:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757276.353836:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757276.353841:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757276.353846:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757276.353849:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757276.353854:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757276.353859:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757276.353862:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757276.353867:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757276.353871:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353875:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757276.353880:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757276.353884:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353888:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757276.353895:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757276.353898:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353902:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757276.353905:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757276.353908:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353916:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757276.353926:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659632000/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757276.353942:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757276.353947:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.353951:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757276.353956:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757276.353964:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757276.353967:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757276.353970:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757276.353975:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757276.353980:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10331 00000100:00080000:3.0:1787757276.353986:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757276.353990:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757276.353993:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757276.353998:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757276.354001:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757276.354004:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757276.354009:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757276.354013:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757276.354016:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757276.354022:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757276.354027:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757276.354031:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757276.354036:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757276.354041:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.354045:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757276.354048:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757276.354054:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.354057:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757276.354065:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757276.354067:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.354071:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757276.354074:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757276.354077:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.354086:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757276.354094:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659632128/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757276.354122:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757276.354126:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757276.354130:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757278.145481:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757278.145489:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757278.145496:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757279.169396:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757279.169401:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757279.169405:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757282.241437:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757282.241445:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757282.241453:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:3.0:1787757283.265441:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757283.265447:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757283.265452:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000001:2.0:1787757286.337486:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757286.337492:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757286.337497:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757286.594015:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757286.594018:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757286.594023:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659632768:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757286.594028:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594030:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757286.594031:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757286.594034:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757286.594036:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757286.594038:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757286.594040:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757286.594044:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757286.594045:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757286.594048:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757286.594049:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594051:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594057:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757286.594059:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594061:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757286.594062:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757286.594064:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757286.594065:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757286.594066:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757286.594068:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757286.594070:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757286.594072:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757286.594074:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659632896:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757286.594077:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757286.594079:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757286.594081:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757286.594082:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757286.594085:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594090:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757286.594091:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:0.0:1787757286.594095:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757286.594108:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757286.594112:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757286.594114:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757286.594116:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594117:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757286.594119:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594123:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305013773 00000100:00000001:0.0:1787757286.594126:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305014465 00000100:00000001:0.0:1787757286.594128:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305013773 00000100:00000001:0.0:1787757286.594130:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305014465 00000100:00000001:0.0:1787757286.594131:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757286.594171:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594173:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757286.594177:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757286/real 1787757286] req@ffff8ffb4a536300 x1874588659632896/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757286.594208:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757286.594210:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757286.594212:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757286.594214:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757286.594215:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594217:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757286.594218:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757286.594221:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659632896/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757286.594228:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594230:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757286.594232:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594233:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757286.594234:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594236:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757286.594246:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757286.594248:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757286.594251:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757286.594252:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757286.594255:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659632896:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757286.594259:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757286.594260:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757286.594261:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757286.594263:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757286.594265:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757286.594266:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757286.594269:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757286.594272:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757286.594273:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757286.594275:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757286.594276:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757286.594279:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594284:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757286.594285:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787757286.594288:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757286.594299:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757286.594303:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757286.594305:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757286.594306:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594308:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757286.594310:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594313:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305013771 00000100:00000001:0.0:1787757286.594315:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305014465 00000100:00000001:0.0:1787757286.594317:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305013771 00000100:00000001:0.0:1787757286.594319:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305014465 00000100:00000001:0.0:1787757286.594320:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757286.594322:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594323:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757286.594326:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757286/real 1787757286] req@ffff8ffb4a537800 x1874588659633024/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757286.594338:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757286.594339:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757286.594341:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757286.594342:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757286.594343:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594345:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757286.594346:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757286.594349:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659633024/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757286.594354:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594356:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757286.594358:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594359:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757286.594360:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594362:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757286.594370:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757286.594372:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757286.594374:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757286.594376:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757286.594379:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757286.594382:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757286.594383:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757286.594385:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757286.594386:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757286.594388:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757286.594389:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757286.594391:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633152:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757286.594394:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757286.594395:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757286.594396:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757286.594398:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757286.594400:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594403:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757286.594404:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787757286.594407:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757286.594415:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757286.594418:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757286.594419:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757286.594421:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594422:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757286.594424:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594426:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305013771 00000100:00000001:0.0:1787757286.594428:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305014465 00000100:00000001:0.0:1787757286.594430:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305013771 00000100:00000001:0.0:1787757286.594432:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305014465 00000100:00000001:0.0:1787757286.594433:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757286.594435:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594436:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757286.594439:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757286/real 1787757286] req@ffff8ffb4a534700 x1874588659633152/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757286.594461:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757286.594463:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757286.594466:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757286.594467:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757286.594469:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594471:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757286.594472:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757286.594477:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659633152/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757286.594486:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594490:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757286.594495:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594498:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757286.594500:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594504:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757286.594522:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757286.594527:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757286.594533:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757286.594538:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757286.594544:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633152:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757286.594551:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594555:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757286.594557:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757286.594560:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757286.594564:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757286.594567:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757286.594569:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757286.594574:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757286.594576:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757286.594580:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757286.594583:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594587:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757286.594588:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757286.594590:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757286.594593:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757286.594594:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757286.594596:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757286.594599:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757286.594600:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757286.594602:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757286.594604:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594621:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757286.594622:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757286.594623:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757286.594624:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757286.594625:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757286.594626:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757286.594627:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757286.594628:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757286.594629:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757286.594631:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594632:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757286.594635:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757286.594637:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757286.594638:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757286.594640:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757287.169336:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757287.169348:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757287.169351:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757287.169359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757287.169364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757287.169370:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757287.169373:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757287.169377:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757287.169381:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757287.361402:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757287.361415:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757287.361427:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757287.361430:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757287.361500:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757287.361506:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757287.361511:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757288.193284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757288.193291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757288.193294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757288.193300:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757288.193305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757288.193309:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757288.193312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757288.193316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757288.193320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757288.385323:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757288.385331:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757288.385338:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757288.385340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757289.217302:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757289.217311:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757289.217314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757289.217321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757289.217326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757289.217331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757289.217334:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757289.217337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757289.217341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757289.409260:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757289.409266:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757289.409271:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757289.409272:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757289.409290:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757289.409293:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757289.409295:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757290.241295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757290.241305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757290.241308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757290.241314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757290.241318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757290.241322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757290.241326:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757290.241329:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757290.241333:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757290.433329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757290.433337:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757290.433347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757290.433351:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757290.433354:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757290.433358:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757290.433362:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757290.433365:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757290.433369:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757290.433373:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757290.433486:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757290.433492:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757290.433497:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757291.265268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757291.265277:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757291.265280:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757291.265287:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757291.265291:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757291.265295:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.265299:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757291.265302:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.265306:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757291.457342:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757291.457350:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757291.457359:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757291.457361:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757291.457453:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757291.457462:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757291.457469:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757291.713328:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757291.713342:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757291.713345:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757291.713349:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757291.713355:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757291.713365:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10346 00000100:00080000:2.0:1787757291.713379:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757291.713383:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757291.713385:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757291.713391:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757291.713394:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757291.713398:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757291.713401:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757291.713404:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757291.713406:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757291.713412:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757291.713415:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757291.713419:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757291.713423:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757291.713427:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713430:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757291.713433:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757291.713446:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757291.713465:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713470:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757291.713473:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757291.713478:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713495:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757291.713507:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659633280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757291.713521:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757291.713526:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713545:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757291.713550:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757291.713554:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757291.713556:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757291.713559:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757291.713562:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757291.713565:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10346 00000100:00080000:2.0:1787757291.713570:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757291.713573:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757291.713576:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757291.713579:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757291.713581:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757291.713585:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757291.713588:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757291.713590:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757291.713593:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757291.713597:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757291.713601:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757291.713602:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757291.713610:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757291.713612:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757291.713617:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757291.713618:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757291.713625:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787757291.713626:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.713631:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713633:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757291.713643:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757291.713644:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757291.713650:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:2.0:1787757291.713652:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.713660:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713660:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757291.713674:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:0.0:1787757291.713677:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659633280:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757291.713678:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713685:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757291.713689:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757291.713692:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757291.713695:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.713701:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787757291.713705:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.713707:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757291.713719:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659633408/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757291.713731:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757291.713738:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757291.713742:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.713743:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713747:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757291.713753:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757291.713757:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757291.713760:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757291.713762:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757291.713765:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757291.713766:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787757291.713772:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10346 00000100:00000001:0.0:1787757291.713774:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00080000:2.0:1787757291.713784:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757291.713786:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757291.713792:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757291.713797:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757291.713800:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757291.713802:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757291.713805:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757291.713808:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757291.713811:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757291.713812:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757291.713816:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757291.713819:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757291.713822:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757291.713826:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757291.713828:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713832:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757291.713835:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757291.713839:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713841:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757291.713848:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757291.713851:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757291.713855:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757291.713857:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757291.713865:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757291.713868:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713876:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.713877:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:2.0:1787757291.713886:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659633536/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757291.713887:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757291.713893:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.713903:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757291.713912:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757291.713914:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.713916:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.713923:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305019585 00000100:00000001:2.0:1787757291.713924:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757291.713934:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757291.713938:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757291.713940:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757291.713942:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757291.713943:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757291.713946:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757291.713954:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10346 00000100:00000001:0.0:1787757291.713955:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305019585 00000100:00080000:2.0:1787757291.713959:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757291.713966:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757291.713967:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757291.713972:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757291.713976:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787757291.713978:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757291.713984:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757291.713986:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757291.713988:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757291.713993:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:2.0:1787757291.713995:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757291.714002:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757291.714004:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787757291.714010:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757291/real 1787757291] req@ffff8ffb47a39880 x1874588659633280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757291.714014:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757291.714025:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757291.714032:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757291.714040:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:0.0:1787757291.714067:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757291.714073:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757291.714079:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:2.0:1787757291.714085:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714087:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757291.714092:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757291.714096:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757291.714098:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757291.714105:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:2.0:1787757291.714107:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757291.714116:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659633280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:2.0:1787757291.714118:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.714129:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714145:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757291.714146:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757291.714155:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714156:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757291.714161:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757291.714171:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757291.714171:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757291.714178:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714241:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757291.714247:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714255:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757291.714259:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757291.714261:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757291.714265:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757291.714277:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659633664/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757291.714278:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757291.714293:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757291.714301:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757291.714302:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757291.714314:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757291.714317:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659633280:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757291.714328:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714332:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757291.714334:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757291.714338:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757291.714345:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757291.714348:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757291.714351:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757291.714356:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757291.714359:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757291.714363:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757291.714365:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714368:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714377:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757291.714380:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714382:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757291.714384:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757291.714387:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757291.714396:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757291.714398:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.714401:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757291.714404:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757291.714407:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757291.714411:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633408:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757291.714416:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757291.714419:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757291.714421:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.714424:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757291.714428:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714437:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757291.714440:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:0.0:1787757291.714445:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757291.714465:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757291.714473:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757291.714476:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757291.714478:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714480:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757291.714484:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714491:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305018894 00000100:00000001:0.0:1787757291.714496:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305019585 00000100:00000001:0.0:1787757291.714501:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305018894 00000100:00000001:0.0:1787757291.714504:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305019585 00000100:00000001:0.0:1787757291.714507:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757291.714510:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714512:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757291.714519:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757291/real 1787757291] req@ffff8ffb47a39500 x1874588659633408/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757291.714532:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757291.714534:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757291.714537:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757291.714539:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757291.714541:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714543:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757291.714545:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757291.714550:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659633408/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757291.714559:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714562:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757291.714565:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714567:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757291.714568:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714570:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757291.714585:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757291.714588:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757291.714592:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757291.714596:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757291.714600:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633408:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757291.714604:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757291.714606:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757291.714608:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.714611:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757291.714613:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757291.714615:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757291.714619:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757291.714623:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757291.714625:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757291.714628:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.714630:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757291.714634:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714640:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757291.714642:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:0.0:1787757291.714647:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757291.714662:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757291.714668:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757291.714670:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757291.714672:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714674:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757291.714678:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714681:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305018891 00000100:00000001:0.0:1787757291.714685:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305019585 00000100:00000001:0.0:1787757291.714687:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305018891 00000100:00000001:0.0:1787757291.714690:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305019585 00000100:00000001:0.0:1787757291.714693:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757291.714696:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714698:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757291.714702:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757291/real 1787757291] req@ffff8ffb47a38000 x1874588659633536/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757291.714712:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757291.714713:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757291.714716:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757291.714718:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757291.714719:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714721:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757291.714723:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757291.714727:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659633536/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757291.714736:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714738:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757291.714742:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714744:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757291.714746:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714748:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757291.714760:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757291.714764:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757291.714767:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757291.714770:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757291.714774:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757291.714778:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757291.714780:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757291.714781:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.714784:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757291.714786:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757291.714788:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757291.714791:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633664:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757291.714795:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757291.714797:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757291.714798:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757291.714800:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757291.714804:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714809:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757291.714811:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757291.714815:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757291.714826:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757291.714831:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757291.714833:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757291.714836:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714838:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757291.714841:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714845:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305018891 00000100:00000001:0.0:1787757291.714848:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305019585 00000100:00000001:0.0:1787757291.714851:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305018891 00000100:00000001:0.0:1787757291.714854:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305019585 00000100:00000001:0.0:1787757291.714857:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757291.714859:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.714861:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757291.714866:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757291/real 1787757291] req@ffff8ffb47a38a80 x1874588659633664/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757291.714875:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757291.714876:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757291.714878:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757291.714880:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757291.714881:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714883:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757291.714885:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757291.714889:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659633664/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757291.714898:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714900:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757291.714903:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714906:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757291.714907:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714909:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757291.714921:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757291.714924:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757291.714927:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757291.714930:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757291.714934:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633664:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757291.714938:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714941:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757291.714943:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757291.714945:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757291.714948:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757291.714950:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757291.714952:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757291.714954:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757291.714956:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757291.714959:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757291.714961:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714963:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757291.714964:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757291.714966:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757291.714968:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757291.714970:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757291.714971:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757291.714973:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757291.714975:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757291.714977:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757291.714979:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714981:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757291.714982:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757291.714984:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757291.714985:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757291.714987:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757291.714988:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757291.714991:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757291.714992:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757291.714994:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757291.714995:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.714997:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757291.715001:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757291.715003:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757291.715005:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757291.715007:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757292.289323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757292.289332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757292.289335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757292.289342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757292.289346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757292.289351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757292.289354:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757292.289358:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757292.289362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757292.481336:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757292.481347:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757292.481358:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757292.481362:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757293.313315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757293.313323:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757293.313326:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757293.313333:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757293.313338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757293.313342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757293.313346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757293.313350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757293.313353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757293.505341:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757293.505351:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757293.505355:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757293.505359:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757293.505365:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757293.505368:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757293.505371:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757293.505374:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757293.505379:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757293.505381:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757293.505408:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757293.505411:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757293.505415:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757294.337295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757294.337304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757294.337307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757294.337313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757294.337318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757294.337323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757294.337327:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757294.337330:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757294.337334:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757294.529308:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757294.529316:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757294.529325:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757294.529328:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757294.529448:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757294.529454:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757294.529459:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757295.361261:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757295.361267:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757295.361268:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757295.361273:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757295.361276:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757295.361279:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757295.361281:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757295.361284:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757295.361286:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757295.553260:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757295.553266:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757295.553271:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757295.553273:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757295.553369:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757295.553374:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757295.553377:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757296.385298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757296.385306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757296.385309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757296.385316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757296.385321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757296.385326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.385329:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757296.385333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.385336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757296.577313:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757296.577321:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757296.577330:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757296.577332:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757296.833304:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757296.833316:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757296.833319:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757296.833323:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757296.833329:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757296.833337:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10351 00000100:00080000:2.0:1787757296.833351:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757296.833355:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757296.833357:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757296.833362:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757296.833365:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757296.833369:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757296.833373:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757296.833375:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757296.833377:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757296.833383:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757296.833386:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757296.833389:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757296.833393:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757296.833397:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833400:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757296.833402:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757296.833416:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757296.833418:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833422:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757296.833425:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757296.833431:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833445:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757296.833457:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659633792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757296.833471:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757296.833477:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833497:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757296.833502:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757296.833506:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757296.833508:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757296.833510:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757296.833514:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757296.833517:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10351 00000100:00080000:2.0:1787757296.833526:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757296.833529:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757296.833531:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757296.833534:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757296.833537:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757296.833541:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757296.833544:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757296.833547:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757296.833550:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757296.833554:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757296.833558:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757296.833560:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757296.833567:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757296.833567:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757296.833576:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.833579:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757296.833580:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757296.833587:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:2.0:1787757296.833591:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757296.833593:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:2.0:1787757296.833617:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.833618:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833622:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.833625:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757296.833628:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757296.833629:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757296.833631:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.833636:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833637:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757296.833641:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757296.833644:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757296.833648:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659633792:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757296.833651:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757296.833660:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659633920/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757296.833669:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757296.833670:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757296.833675:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.833679:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787757296.833679:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.833687:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757296.833687:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757296.833696:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757296.833697:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757296.833698:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757296.833703:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757296.833705:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757296.833709:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757296.833709:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10351 00000100:00080000:2.0:1787757296.833715:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757296.833718:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757296.833719:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757296.833721:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757296.833722:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757296.833724:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757296.833726:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757296.833728:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757296.833729:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757296.833731:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757296.833734:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757296.833734:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757296.833736:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757296.833740:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 02000000:00000001:2.0:1787757296.833741:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757296.833746:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.833748:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757296.833749:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757296.833754:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757296.833756:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833758:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757296.833762:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757296.833763:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833764:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757296.833766:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757296.833767:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833771:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757296.833775:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659634048/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757296.833782:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757296.833784:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833786:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757296.833788:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757296.833795:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787757296.833795:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757296.833797:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757296.833801:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757296.833804:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757296.833808:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10351 00000100:00000001:0.0:1787757296.833811:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:2.0:1787757296.833812:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757296.833817:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757296.833819:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757296.833819:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757296.833824:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757296.833825:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757296.833845:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757296.833848:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:2.0:1787757296.833850:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757296.833857:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757296.833861:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757296.833862:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833867:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757296.833872:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305024705 00000100:00080000:2.0:1787757296.833876:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757296.833883:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757296.833886:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757296.833890:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757296.833891:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757296.833893:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833901:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757296.833905:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305024705 00000008:00100000:2.0:1787757296.833906:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757296.833917:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833920:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.833921:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757296.833927:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757296.833932:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.833934:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757296.833935:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757296.833937:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.833944:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757296.833945:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.833996:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757296.834001:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659634176/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757296.834002:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757296.834014:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757296/real 1787757296] req@ffff8ffb47a38380 x1874588659633792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757296.834014:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757296.834019:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757296.834026:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757296.834037:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757296.834057:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757296.834063:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757296.834066:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757296.834071:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834074:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757296.834077:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757296.834084:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659633792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757296.834096:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834100:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757296.834104:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834107:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757296.834109:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834112:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757296.834115:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757296.834119:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757296.834124:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757296.834127:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757296.834132:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659633792:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757296.834138:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834141:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757296.834143:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757296.834146:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757296.834150:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757296.834153:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757296.834156:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757296.834162:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757296.834164:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757296.834168:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757296.834170:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834173:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834181:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757296.834183:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834186:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757296.834189:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757296.834191:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757296.834193:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757296.834195:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.834238:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757296.834241:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757296.834244:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757296.834248:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633920:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757296.834253:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757296.834256:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757296.834258:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.834261:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757296.834266:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834274:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757296.834276:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757296.834282:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757296.834302:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757296.834310:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757296.834314:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757296.834317:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834321:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757296.834327:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834335:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305024014 00000100:00000001:0.0:1787757296.834342:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305024705 00000100:00000001:0.0:1787757296.834346:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305024014 00000100:00000001:0.0:1787757296.834352:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305024705 00000100:00000001:0.0:1787757296.834355:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757296.834360:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834364:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757296.834374:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757296/real 1787757296] req@ffff8ffb47a3a680 x1874588659633920/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757296.834393:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757296.834396:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757296.834402:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757296.834405:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757296.834408:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834412:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757296.834415:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757296.834422:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659633920/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757296.834435:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834439:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757296.834443:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834447:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757296.834449:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834452:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757296.834474:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757296.834479:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757296.834485:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757296.834490:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757296.834496:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659633920:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757296.834504:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757296.834507:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757296.834510:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.834514:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757296.834517:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757296.834531:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757296.834537:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659634048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757296.834543:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757296.834548:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757296.834551:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.834555:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757296.834562:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834573:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757296.834576:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757296.834583:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757296.834620:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757296.834626:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757296.834628:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757296.834630:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834631:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757296.834634:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834637:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305024011 00000100:00000001:0.0:1787757296.834640:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305024705 00000100:00000001:0.0:1787757296.834642:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305024011 00000100:00000001:0.0:1787757296.834644:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305024705 00000100:00000001:0.0:1787757296.834646:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757296.834648:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834650:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757296.834654:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757296/real 1787757296] req@ffff8ffb47a3b480 x1874588659634048/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757296.834662:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757296.834663:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757296.834665:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757296.834667:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757296.834669:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834671:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757296.834673:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757296.834678:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659634048/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757296.834688:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834690:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757296.834694:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834696:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757296.834698:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834700:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757296.834715:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757296.834719:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757296.834724:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757296.834727:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757296.834732:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659634048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757296.834737:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757296.834739:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757296.834741:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.834743:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757296.834745:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757296.834747:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757296.834750:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659634176:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757296.834753:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757296.834754:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757296.834756:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757296.834758:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757296.834761:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834766:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757296.834767:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757296.834770:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757296.834782:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757296.834787:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757296.834789:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757296.834790:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834792:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757296.834794:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834797:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305024011 00000100:00000001:0.0:1787757296.834799:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305024705 00000100:00000001:0.0:1787757296.834801:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305024011 00000100:00000001:0.0:1787757296.834803:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305024705 00000100:00000001:0.0:1787757296.834805:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757296.834807:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.834808:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757296.834812:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757296/real 1787757296] req@ffff8ffb47a39180 x1874588659634176/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757296.834818:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757296.834819:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757296.834821:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757296.834822:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757296.834824:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834825:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757296.834827:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757296.834848:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659634176/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757296.834865:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834868:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757296.834871:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834873:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757296.834875:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834877:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757296.834890:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757296.834893:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757296.834896:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757296.834899:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757296.834902:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659634176:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757296.834907:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834910:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757296.834911:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757296.834914:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757296.834916:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757296.834918:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757296.834920:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757296.834923:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757296.834925:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757296.834929:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757296.834931:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834933:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757296.834934:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757296.834936:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757296.834939:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757296.834941:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757296.834942:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757296.834944:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757296.834945:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757296.834948:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757296.834950:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.834951:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757296.834987:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757296.834990:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757296.834992:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757296.834994:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757296.834996:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757296.835009:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757296.835011:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757296.835014:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757296.835016:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.835019:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757296.835024:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757296.835026:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757296.835029:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757296.835031:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757297.409322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757297.409331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757297.409334:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757297.409342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757297.409347:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757297.409351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757297.409354:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757297.409357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757297.409361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757297.601400:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757297.601412:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757297.601424:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757297.601428:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757297.601469:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757297.601473:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757297.601477:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757298.433288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757298.433297:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757298.433300:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757298.433306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757298.433310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757298.433315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757298.433319:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757298.433322:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757298.433325:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757298.625301:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757298.625313:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757298.625323:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757298.625326:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757298.625423:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757298.625431:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757298.625438:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757299.457335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757299.457344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757299.457347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757299.457353:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757299.457358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757299.457363:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757299.457366:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757299.457369:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757299.457373:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757299.649304:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757299.649313:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757299.649321:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757299.649324:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757299.649422:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757299.649431:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757299.649438:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757300.481286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757300.481296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757300.481299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757300.481306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757300.481310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757300.481316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757300.481320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757300.481325:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757300.481331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757300.673294:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757300.673305:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757300.673317:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757300.673323:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757300.673327:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757300.673333:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757300.673339:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757300.673342:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757300.673346:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757300.673351:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.505288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757301.505298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757301.505301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757301.505310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757301.505315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757301.505319:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757301.505322:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757301.505326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757301.505329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757301.697335:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757301.697342:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757301.697350:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757301.697352:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757301.697377:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757301.697381:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757301.697384:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757301.953358:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757301.953371:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787757301.953374:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757301.953379:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757301.953385:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757301.953394:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10356 00000100:00080000:0.0:1787757301.953408:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757301.953411:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757301.953414:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757301.953419:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757301.953423:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757301.953427:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787757301.953430:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787757301.953433:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757301.953435:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757301.953441:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757301.953444:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757301.953447:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757301.953451:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757301.953455:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953459:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757301.953463:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757301.953481:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757301.953485:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953505:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757301.953512:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757301.953519:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953539:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757301.953556:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ce00 x1874588659634304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757301.953576:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757301.953585:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953613:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757301.953620:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757301.953627:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787757301.953630:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757301.953633:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757301.953638:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757301.953642:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10356 00000100:00080000:0.0:1787757301.953649:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757301.953652:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757301.953654:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757301.953658:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757301.953661:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757301.953665:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787757301.953668:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757301.953671:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757301.953673:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757301.953678:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757301.953681:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757301.953685:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757301.953697:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757301.953706:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953710:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757301.953720:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757301.953727:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953737:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757301.953745:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757301.953750:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953753:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757301.953758:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757301.953767:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953782:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757301.953796:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790c380 x1874588659634432/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757301.953820:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757301.953828:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953835:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757301.953843:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757301.953848:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757301.953850:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757301.953889:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757301.953899:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757301.953906:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10356 00000100:00080000:0.0:1787757301.953917:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757301.953920:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757301.953922:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757301.953926:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757301.953928:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757301.953931:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787757301.953934:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757301.953938:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757301.953939:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757301.953944:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757301.953948:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757301.953952:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757301.953956:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757301.953959:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953962:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757301.953965:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757301.953971:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953976:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757301.953988:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757301.953991:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.953994:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757301.953997:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757301.954001:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.954012:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757301.954028:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f100 x1874588659634560/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757301.954051:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757301.954055:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.954059:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757301.954064:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757301.954071:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757301.954075:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757301.954081:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757301.954087:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757301.954096:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10356 00000100:00080000:0.0:1787757301.954107:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757301.954113:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757301.954118:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757301.954125:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757301.954128:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757301.954134:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:0.0:1787757301.954143:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757301.954151:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757301.954155:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757301.954169:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757301.954172:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757301.954179:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757301.954230:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757301.954233:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.954236:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757301.954239:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757301.954242:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.954244:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757301.954250:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757301.954255:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.954258:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757301.954261:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757301.954269:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.954277:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757301.954296:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790d880 x1874588659634688/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757301.954313:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757301.954319:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757301.954323:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757302.529304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757302.529314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757302.529317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757302.529324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757302.529329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757302.529334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757302.529337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757302.529340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757302.529343:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757302.721346:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757302.721356:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757302.721367:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757302.721370:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757302.721442:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757302.721448:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757302.721453:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757303.553288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757303.553296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757303.553299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757303.553305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757303.553309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757303.553314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757303.553317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757303.553320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757303.553323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757303.745296:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757303.745317:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757303.745320:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757303.745324:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757303.745329:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757303.745332:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757303.745333:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757303.745335:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757303.745339:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757303.745340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757303.745401:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757303.745405:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757303.745408:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757304.577280:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757304.577289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757304.577292:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757304.577298:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757304.577303:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757304.577307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757304.577311:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757304.577314:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757304.577318:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757304.769341:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757304.769349:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757304.769356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757304.769358:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757305.601300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757305.601318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757305.601321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757305.601328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757305.601332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757305.601335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757305.601338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757305.601341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757305.601344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757305.793313:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757305.793321:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757305.793329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757305.793332:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757305.793361:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757305.793365:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757305.793369:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757306.625311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757306.625318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757306.625320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757306.625325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757306.625329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757306.625333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757306.625336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757306.625338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757306.625341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757306.817329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757306.817337:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757306.817343:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757306.817344:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757306.817429:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757306.817434:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757306.817439:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757307.073383:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757307.073395:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757307.073399:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757307.073404:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757307.073410:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757307.073432:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10361 00000100:00080000:3.0:1787757307.073447:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757307.073451:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757307.073455:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757307.073461:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757307.073465:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757307.073469:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757307.073473:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757307.073477:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757307.073479:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757307.073487:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757307.073490:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757307.073494:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757307.073499:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757307.073503:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.073506:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757307.073509:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757307.073521:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757307.073523:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.073527:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757307.073530:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757307.073536:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.073551:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757307.073564:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659634816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757307.073580:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757307.073587:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.073611:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757307.073616:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757307.073622:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757307.073624:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757307.073626:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757307.073630:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757307.073634:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10361 00000100:00080000:3.0:1787757307.073640:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757307.073644:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757307.073647:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757307.073650:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757307.073652:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757307.073655:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757307.073659:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757307.073662:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757307.073664:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757307.073668:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757307.073671:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757307.073673:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757307.073678:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757307.073680:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757307.073688:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.073689:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757307.073696:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757307.073698:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757307.073704:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:3.0:1787757307.073705:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757307.073707:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757307.073717:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.073721:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757307.073723:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.073729:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757307.073732:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757307.073735:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757307.073737:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.073743:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757307.073744:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757307.073747:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.073753:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757307.073763:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659634816:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757307.073797:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757307.073804:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757307.073808:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.073814:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757307.073823:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.073847:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757307.073851:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787757307.073858:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757307.073913:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1787757307.073922:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659634944/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757307.073930:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757307.073938:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757307.073941:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.073943:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757307.073950:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.073955:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305034945 00000100:00000001:2.0:1787757307.073985:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757307.074002:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305034945 00000100:00000001:2.0:1787757307.074012:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757307.074019:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757307.074025:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074030:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757307.074043:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757307/real 1787757307] req@ffff8ffb45c30000 x1874588659634816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757307.074067:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757307.074071:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757307.074081:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757307.074086:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757307.074090:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074094:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757307.074101:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757307.074106:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757307.074114:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757307.074119:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659634816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757307.074124:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757307.074133:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757307.074146:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757307.074152:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757307.074163:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074171:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757307.074177:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074181:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757307.074227:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074232:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757307.074237:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757307.074243:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757307.074251:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757307.074256:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757307.074265:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659634816:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757307.074273:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757307.074276:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074288:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757307.074289:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757307.074291:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757307.074299:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00080000:3.0:1787757307.074306:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10361 02000000:00000001:2.0:1787757307.074309:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757307.074317:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757307.074320:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00080000:3.0:1787757307.074322:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:2.0:1787757307.074327:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787757307.074332:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787757307.074334:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757307.074340:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757307.074342:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000020:00000001:3.0:1787757307.074347:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757307.074349:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757307.074351:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757307.074356:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757307.074358:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757307.074366:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757307.074368:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757307.074370:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757307.074372:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757307.074377:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757307.074384:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757307.074386:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757307.074395:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074396:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757307.074400:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757307.074405:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757307.074408:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757307.074412:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757307.074415:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074421:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074421:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757307.074427:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787757307.074433:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:3.0:1787757307.074434:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074442:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.074444:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757307.074450:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757307.074454:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757307.074455:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074465:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757307.074468:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:2.0:1787757307.074476:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659634944:192.168.201.118@tcp:38:kworker.0 02000000:00000001:3.0:1787757307.074477:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074489:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074492:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757307.074501:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:3.0:1787757307.074503:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659635072/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757307.074506:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.074515:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757307.074524:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074526:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757307.074534:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074540:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074542:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757307.074544:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757307.074547:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00080000:3.0:1787757307.074560:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757307.074562:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757307.074565:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757307.074570:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757307.074574:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757307.074579:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10361 00000100:00080000:3.0:1787757307.074586:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757307.074589:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757307.074593:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:2.0:1787757307.074595:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787757307.074599:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757307.074604:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757307.074607:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757307.074608:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:3.0:1787757307.074615:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757307.074618:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757307.074620:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757307.074624:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074626:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757307.074631:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757307.074638:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757307.074641:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074643:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074650:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305034254 02000000:00000001:3.0:1787757307.074650:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757307.074658:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757307.074660:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305034945 02000000:00000001:3.0:1787757307.074665:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074668:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305034254 00000100:00000001:3.0:1787757307.074670:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757307.074676:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305034945 00000008:00100000:3.0:1787757307.074678:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757307.074687:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787757307.074688:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074700:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074700:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074711:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787757307.074714:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757307.074718:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074724:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:2.0:1787757307.074725:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757307/real 1787757307] req@ffff8ffb45c30700 x1874588659634944/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757307.074726:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757307.074732:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074742:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757307.074754:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00100000:3.0:1787757307.074756:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659635200/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:2.0:1787757307.074760:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757307.074770:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757307.074804:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757307.074811:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757307.074813:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757307.074816:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757307.074828:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074833:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757307.074840:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757307.074854:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659634944/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757307.074872:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074877:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757307.074882:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074884:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757307.074886:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074888:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757307.074904:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757307.074908:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757307.074913:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757307.074917:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757307.074921:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659634944:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757307.074927:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074930:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757307.074932:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757307.074934:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757307.074938:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757307.074940:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757307.074942:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757307.074945:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757307.074947:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757307.074951:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757307.074952:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074955:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.074960:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757307.074961:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.074964:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757307.074966:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757307.074968:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757307.074970:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757307.074972:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.074975:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757307.074978:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757307.074981:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757307.074985:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757307.074989:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757307.074992:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757307.074994:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.074996:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757307.075001:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075009:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757307.075011:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787757307.075016:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757307.075035:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757307.075041:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757307.075061:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757307.075063:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075064:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757307.075067:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075071:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305034251 00000100:00000001:2.0:1787757307.075074:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305034945 00000100:00000001:2.0:1787757307.075076:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305034251 00000100:00000001:2.0:1787757307.075078:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305034945 00000100:00000001:2.0:1787757307.075080:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757307.075082:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075084:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757307.075088:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757307/real 1787757307] req@ffff8ffb45c32a00 x1874588659635072/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757307.075097:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757307.075098:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757307.075100:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757307.075102:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757307.075103:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075105:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757307.075106:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757307.075110:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659635072/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757307.075117:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075119:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757307.075121:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075122:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757307.075124:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075125:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757307.075135:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757307.075137:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757307.075140:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757307.075142:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757307.075144:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757307.075148:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757307.075149:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757307.075150:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.075152:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757307.075154:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757307.075156:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757307.075159:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757307.075161:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757307.075163:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757307.075165:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757307.075167:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757307.075169:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075173:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757307.075175:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787757307.075178:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757307.075187:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757307.075191:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757307.075193:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757307.075195:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075196:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757307.075231:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075235:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305034252 00000100:00000001:2.0:1787757307.075238:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305034945 00000100:00000001:2.0:1787757307.075240:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305034252 00000100:00000001:2.0:1787757307.075242:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305034945 00000100:00000001:2.0:1787757307.075244:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757307.075246:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075249:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757307.075253:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757307/real 1787757307] req@ffff8ffb45c33800 x1874588659635200/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757307.075261:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757307.075263:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757307.075267:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757307.075269:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757307.075271:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075273:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757307.075274:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757307.075280:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659635200/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757307.075287:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075290:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757307.075294:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075296:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757307.075297:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075299:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757307.075314:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757307.075317:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757307.075320:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757307.075322:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757307.075324:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757307.075328:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075330:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757307.075331:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757307.075333:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757307.075335:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757307.075336:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757307.075338:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757307.075340:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757307.075341:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757307.075343:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757307.075345:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075346:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757307.075347:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757307.075348:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757307.075350:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757307.075351:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757307.075353:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757307.075354:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757307.075355:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757307.075356:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757307.075357:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075359:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757307.075362:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757307.075363:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757307.075365:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757307.075367:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757307.649236:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757307.649243:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757307.649245:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757307.649248:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757307.649251:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757307.649254:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757307.649256:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757307.649258:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757307.649260:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757307.841301:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757307.841307:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757307.841312:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757307.841314:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757307.841332:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757307.841336:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757307.841339:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757308.673327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757308.673335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757308.673337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757308.673343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757308.673347:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757308.673350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757308.673353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757308.673356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757308.673359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757308.865335:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757308.865346:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757308.865353:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757308.865371:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757309.697320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757309.697326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757309.697329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757309.697334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757309.697337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757309.697341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757309.697343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757309.697346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757309.697349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757309.889350:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757309.889358:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757309.889364:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757309.889366:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757309.889402:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757309.889405:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757309.889409:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757310.721278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757310.721287:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757310.721291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757310.721298:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757310.721304:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757310.721310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757310.721315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757310.721319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757310.721325:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757310.913373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757310.913383:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757310.913391:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757310.913396:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757310.913412:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757310.913416:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757310.913420:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757310.913424:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757310.913426:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757310.913429:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757310.913541:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757310.913550:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757310.913556:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757311.745314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757311.745324:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757311.745326:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757311.745332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757311.745337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757311.745341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757311.745345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757311.745348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757311.745351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757311.937277:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757311.937282:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757311.937287:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757311.937289:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757311.937340:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757311.937344:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757311.937348:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757312.193257:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757312.193265:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757312.193268:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757312.193271:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757312.193274:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757312.193280:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10366 00000100:00080000:3.0:1787757312.193289:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757312.193292:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757312.193294:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757312.193297:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757312.193299:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757312.193302:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757312.193304:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757312.193305:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757312.193306:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757312.193310:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757312.193311:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757312.193313:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757312.193316:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757312.193318:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193320:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757312.193322:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757312.193332:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757312.193333:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193336:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757312.193338:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757312.193341:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193352:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757312.193359:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659635328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757312.193367:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757312.193370:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193382:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757312.193385:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757312.193387:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757312.193388:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757312.193389:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757312.193391:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757312.193393:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10366 00000100:00080000:3.0:1787757312.193395:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757312.193396:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757312.193397:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757312.193398:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757312.193399:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757312.193400:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757312.193402:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757312.193404:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757312.193405:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757312.193407:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757312.193409:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757312.193411:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757312.193415:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757312.193417:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193419:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757312.193421:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787757312.193423:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757312.193427:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:3.0:1787757312.193428:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193431:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193433:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757312.193435:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757312.193437:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193438:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757312.193439:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757312.193442:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757312.193444:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757312.193444:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193450:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757312.193452:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:3.0:1787757312.193456:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659635456/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757312.193458:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757312.193466:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193467:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757312.193470:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193473:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757312.193475:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659635328:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757312.193476:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757312.193483:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757312.193486:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757312.193488:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757312.193488:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757312.193493:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:2.0:1787757312.193495:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787757312.193496:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10366 00000100:00080000:3.0:1787757312.193500:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:2.0:1787757312.193501:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193502:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757312.193504:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757312.193507:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787757312.193508:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757312.193511:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757312.193513:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757312.193514:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757312.193516:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193516:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757312.193518:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757312.193522:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757312.193524:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757312.193525:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757312.193527:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757312.193528:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193529:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757312.193531:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757312.193533:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193534:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193536:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757312.193537:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757312.193540:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 02000000:00000001:3.0:1787757312.193540:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193543:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757312.193545:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757312.193548:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193549:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757312.193552:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757312.193556:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659635584/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757312.193561:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757312.193562:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193564:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757312.193565:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757312.193568:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757312.193569:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757312.193570:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757312.193572:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757312.193573:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10366 00000100:00080000:3.0:1787757312.193578:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757312.193580:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757312.193581:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757312.193582:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757312.193583:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757312.193584:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757312.193586:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757312.193590:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000800:00000100:2.0:1787757312.193592:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757312.193592:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757312.193599:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757312.193601:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757312.193602:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757312.193604:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757312.193606:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193607:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757312.193608:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757312.193610:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757312.193613:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193614:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757312.193615:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193620:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757312.193621:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757312.193625:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193626:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757312.193627:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757312.193630:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757312.193631:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193634:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193636:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757312.193638:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659635712/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757312.193642:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305040065 00000100:00000001:3.0:1787757312.193645:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757312.193648:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757312.193649:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193655:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757312.193663:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305040065 00000100:00000001:2.0:1787757312.193668:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757312.193673:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757312.193676:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193679:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757312.193699:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757312/real 1787757312] req@ffff8ffb45c31180 x1874588659635328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757312.193711:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757312.193714:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757312.193719:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757312.193721:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757312.193724:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193726:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757312.193728:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757312.193733:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659635328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757312.193741:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193744:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757312.193747:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193749:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757312.193750:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193753:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757312.193755:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757312.193759:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757312.193762:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757312.193765:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757312.193770:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659635328:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757312.193774:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193777:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757312.193779:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757312.193781:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757312.193787:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757312.193790:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757312.193791:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757312.193795:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757312.193797:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757312.193800:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757312.193801:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193804:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193812:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757312.193814:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193817:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757312.193818:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757312.193820:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757312.193821:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757312.193823:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757312.193825:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757312.193827:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757312.193829:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757312.193832:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635456:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757312.193836:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757312.193838:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757312.193839:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757312.193841:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757312.193844:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193851:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757312.193854:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:2.0:1787757312.193858:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757312.193871:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757312.193876:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757312.193879:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757312.193880:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193882:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757312.193885:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193889:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305039373 00000100:00000001:2.0:1787757312.193893:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305040065 00000100:00000001:2.0:1787757312.193895:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305039373 00000100:00000001:2.0:1787757312.193897:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305040065 00000100:00000001:2.0:1787757312.193899:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757312.193901:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.193903:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757312.193908:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757312/real 1787757312] req@ffff8ffb45c33480 x1874588659635456/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757312.193917:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757312.193918:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757312.193921:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757312.193923:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757312.193924:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193926:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757312.193928:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757312.193932:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659635456/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757312.193938:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193940:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757312.193942:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193944:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757312.193945:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.193947:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757312.193961:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757312.193964:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757312.193967:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757312.193969:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757312.193983:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635456:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757312.193987:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757312.193989:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757312.193990:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757312.193992:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757312.193994:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757312.193996:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757312.193999:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757312.194002:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757312.194004:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757312.194006:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757312.194007:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757312.194010:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194017:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757312.194018:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:2.0:1787757312.194022:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757312.194035:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757312.194040:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757312.194042:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757312.194043:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194045:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757312.194048:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194051:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305039370 00000100:00000001:2.0:1787757312.194054:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305040065 00000100:00000001:2.0:1787757312.194056:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305039370 00000100:00000001:2.0:1787757312.194058:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305040065 00000100:00000001:2.0:1787757312.194060:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757312.194062:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194064:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757312.194069:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757312/real 1787757312] req@ffff8ffb45c31500 x1874588659635584/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757312.194077:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757312.194078:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757312.194081:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757312.194083:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757312.194084:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194086:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757312.194087:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757312.194091:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659635584/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757312.194099:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194102:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757312.194104:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194105:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757312.194107:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194108:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757312.194120:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757312.194123:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757312.194127:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757312.194130:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757312.194135:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757312.194138:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757312.194140:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757312.194141:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757312.194143:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757312.194145:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757312.194147:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757312.194150:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757312.194153:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757312.194155:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757312.194156:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757312.194158:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757312.194161:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194166:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757312.194168:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787757312.194171:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757312.194213:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757312.194219:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757312.194222:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757312.194223:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194225:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757312.194228:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194231:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305039371 00000100:00000001:2.0:1787757312.194235:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305040065 00000100:00000001:2.0:1787757312.194237:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305039371 00000100:00000001:2.0:1787757312.194239:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305040065 00000100:00000001:2.0:1787757312.194241:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757312.194243:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194245:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757312.194250:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757312/real 1787757312] req@ffff8ffb45c33800 x1874588659635712/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757312.194258:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757312.194260:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757312.194262:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757312.194264:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757312.194266:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194268:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757312.194269:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757312.194273:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659635712/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757312.194281:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194284:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757312.194286:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194288:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757312.194289:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194292:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757312.194304:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757312.194307:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757312.194310:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757312.194313:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757312.194316:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757312.194320:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194322:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757312.194323:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757312.194325:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757312.194327:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757312.194329:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757312.194330:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757312.194333:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757312.194334:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757312.194336:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757312.194338:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194340:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757312.194341:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757312.194343:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757312.194345:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757312.194346:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757312.194347:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757312.194349:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757312.194350:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757312.194352:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757312.194353:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194355:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757312.194356:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757312.194357:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757312.194359:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757312.194361:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757312.194362:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757312.194364:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757312.194365:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757312.194366:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757312.194367:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194369:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757312.194373:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757312.194374:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757312.194376:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757312.194378:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757312.769242:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757312.769249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757312.769251:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757312.769255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757312.769257:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757312.769260:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757312.769262:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757312.769264:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757312.769266:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757312.961293:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757312.961300:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757312.961306:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757312.961307:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757313.793320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757313.793332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757313.793336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757313.793344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757313.793349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757313.793354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757313.793357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757313.793360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757313.793364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757313.985439:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757313.985456:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757313.985461:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757313.985501:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757313.985511:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757313.985515:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757313.985519:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757313.985524:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757313.985532:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757313.985535:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757313.985598:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757313.985603:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757313.985609:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757314.817286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757314.817296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757314.817299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757314.817307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757314.817314:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757314.817321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757314.817326:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757314.817331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757314.817335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757315.009453:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757315.009461:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757315.009467:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757315.009469:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757315.009596:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757315.009604:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757315.009612:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757315.841277:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757315.841285:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757315.841288:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757315.841293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757315.841298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757315.841303:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757315.841306:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757315.841309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757315.841311:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757316.033301:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757316.033307:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757316.033312:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757316.033313:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757316.033402:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757316.033405:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757316.033408:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757316.865351:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757316.865359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757316.865362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757316.865368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757316.865373:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757316.865378:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757316.865381:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757316.865384:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757316.865388:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757317.057334:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757317.057342:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757317.057351:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757317.057353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757317.313297:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757317.313315:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757317.313318:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757317.313323:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757317.313329:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757317.313338:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10372 00000100:00080000:3.0:1787757317.313352:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757317.313356:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757317.313359:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757317.313364:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757317.313369:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757317.313373:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757317.313376:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757317.313379:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757317.313381:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757317.313402:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757317.313404:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757317.313406:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757317.313409:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757317.313412:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313413:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757317.313416:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757317.313427:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757317.313429:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313431:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757317.313434:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757317.313438:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313449:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757317.313458:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659635840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757317.313467:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757317.313471:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313488:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757317.313491:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757317.313494:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757317.313495:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757317.313497:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757317.313499:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757317.313501:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10372 00000100:00080000:3.0:1787757317.313504:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757317.313506:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757317.313507:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757317.313509:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757317.313511:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757317.313512:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757317.313514:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757317.313516:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757317.313517:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757317.313519:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757317.313521:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757317.313522:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757317.313527:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757317.313529:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313530:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757317.313535:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757317.313538:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313539:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757317.313542:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757317.313543:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313545:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757317.313546:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757317.313547:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313551:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757317.313555:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659635968/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757317.313562:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757317.313564:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313566:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757317.313568:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757317.313571:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757317.313572:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757317.313574:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757317.313576:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757317.313579:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10372 00000100:00080000:3.0:1787757317.313582:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757317.313584:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757317.313586:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757317.313588:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757317.313590:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757317.313617:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757317.313621:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757317.313624:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757317.313626:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757317.313627:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787757317.313631:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:3.0:1787757317.313633:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757317.313635:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787757317.313640:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757317.313641:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313647:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:2.0:1787757317.313649:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659635840:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757317.313651:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757317.313660:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757317.313665:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787757317.313666:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757317.313671:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313673:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757317.313675:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757317.313677:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757317.313681:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757317.313682:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757317.313688:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313690:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757317.313694:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757317.313696:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313698:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.313702:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757317.313703:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757317.313706:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 02000000:00000001:3.0:1787757317.313706:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313712:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757317.313714:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757317.313715:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757317.313718:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313725:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757317.313730:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659636096/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757317.313737:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757317.313739:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313743:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757317.313745:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757317.313749:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757317.313750:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757317.313753:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787757317.313754:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757317.313757:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757317.313762:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10372 00000100:00080000:3.0:1787757317.313766:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757317.313769:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757317.313769:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757317.313770:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757317.313775:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:3.0:1787757317.313775:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757317.313777:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757317.313780:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757317.313781:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757317.313785:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:3.0:1787757317.313786:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757317.313789:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757317.313791:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757317.313792:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757317.313795:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757317.313799:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305045185 00000100:00000001:3.0:1787757317.313800:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757317.313804:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757317.313808:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757317.313810:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.313812:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757317.313812:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757317.313814:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757317.313818:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757317.313820:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.313822:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305045185 02000000:00000001:3.0:1787757317.313825:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757317.313828:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.313829:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757317.313830:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757317.313833:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757317.313834:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757317.313836:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.313857:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.313862:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757317.313871:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757317/real 1787757317] req@ffff8ffb45c32a00 x1874588659635840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757317.313878:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757317.313894:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659636224/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:2.0:1787757317.313895:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757317.313902:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757317.313909:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757317.313913:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757317.313916:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757317.313919:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.313921:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757317.313927:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.313928:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757317.313935:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757317.313946:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659635840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757317.313959:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.313964:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757317.313969:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.313971:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757317.313972:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.313975:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757317.313978:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757317.313982:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757317.313987:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757317.313990:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757317.313995:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659635840:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757317.314001:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757317.314003:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757317.314005:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757317.314007:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757317.314010:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757317.314013:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757317.314016:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635968:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757317.314020:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757317.314023:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757317.314025:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757317.314027:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757317.314031:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314038:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757317.314039:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787757317.314045:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757317.314060:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757317.314066:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757317.314068:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757317.314070:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314072:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757317.314076:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314081:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305044493 00000100:00000001:2.0:1787757317.314085:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305045185 00000100:00000001:2.0:1787757317.314088:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305044493 00000100:00000001:2.0:1787757317.314091:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305045185 00000100:00000001:2.0:1787757317.314094:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757317.314097:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314099:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757317.314104:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757317/real 1787757317] req@ffff8ffb45c30700 x1874588659635968/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757317.314114:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757317.314115:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757317.314118:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757317.314146:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757317.314147:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314149:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757317.314150:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757317.314153:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659635968/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757317.314158:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314160:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757317.314162:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314164:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757317.314165:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314166:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757317.314177:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757317.314179:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757317.314181:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757317.314183:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757317.314185:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659635968:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757317.314189:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314191:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757317.314193:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757317.314211:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757317.314215:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757317.314218:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757317.314221:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757317.314225:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757317.314227:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757317.314231:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757317.314233:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314235:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757317.314236:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757317.314285:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757317.314289:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757317.314290:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757317.314292:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757317.314294:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757317.314296:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757317.314298:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757317.314301:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314304:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314312:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757317.314315:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314317:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757317.314320:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757317.314322:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757317.314324:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757317.314325:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757317.314329:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757317.314331:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757317.314333:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757317.314337:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659636096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757317.314342:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757317.314344:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757317.314347:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757317.314349:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757317.314353:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314361:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757317.314363:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787757317.314368:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757317.314399:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757317.314403:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757317.314405:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757317.314406:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314407:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757317.314410:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314413:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305044491 00000100:00000001:2.0:1787757317.314415:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305045185 00000100:00000001:2.0:1787757317.314417:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305044491 00000100:00000001:2.0:1787757317.314419:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305045185 00000100:00000001:2.0:1787757317.314421:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757317.314422:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314424:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757317.314427:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757317/real 1787757317] req@ffff8ffb45c30000 x1874588659636096/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757317.314433:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757317.314434:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757317.314435:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757317.314437:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757317.314438:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314439:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757317.314441:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757317.314443:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659636096/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757317.314449:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314450:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757317.314452:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314453:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757317.314454:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314456:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757317.314465:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757317.314467:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757317.314470:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757317.314471:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757317.314474:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659636096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757317.314477:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757317.314478:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757317.314479:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757317.314481:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757317.314482:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757317.314483:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757317.314485:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659636224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757317.314488:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757317.314489:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757317.314490:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757317.314492:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757317.314493:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314497:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757317.314498:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:2.0:1787757317.314501:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757317.314509:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757317.314512:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757317.314513:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757317.314515:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314516:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757317.314518:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314521:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305044491 00000100:00000001:2.0:1787757317.314523:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305045185 00000100:00000001:2.0:1787757317.314525:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305044491 00000100:00000001:2.0:1787757317.314527:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305045185 00000100:00000001:2.0:1787757317.314529:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757317.314530:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314532:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757317.314535:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757317/real 1787757317] req@ffff8ffb45c31c00 x1874588659636224/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757317.314541:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757317.314542:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757317.314544:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757317.314546:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757317.314548:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314550:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757317.314552:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757317.314557:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659636224/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757317.314566:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314569:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757317.314572:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314574:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757317.314575:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314578:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757317.314609:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757317.314614:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757317.314619:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757317.314621:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757317.314626:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659636224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757317.314630:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314632:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757317.314633:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757317.314634:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757317.314637:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757317.314638:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757317.314639:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757317.314641:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757317.314642:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757317.314644:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757317.314645:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314647:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757317.314648:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757317.314649:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757317.314650:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757317.314651:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757317.314652:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757317.314653:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757317.314654:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757317.314656:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757317.314656:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314658:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757317.314660:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757317.314662:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757317.314663:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757317.314664:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757317.889321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757317.889333:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757317.889337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757317.889346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757317.889353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757317.889358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757317.889362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757317.889365:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757317.889368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757318.081338:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757318.081347:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757318.081357:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757318.081360:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757318.081392:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757318.081397:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757318.081402:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757318.913323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757318.913329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757318.913332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757318.913337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757318.913341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757318.913345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757318.913348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757318.913351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757318.913354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757319.105273:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757319.105279:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757319.105283:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757319.105285:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757319.105344:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757319.105348:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757319.105352:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757319.937296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757319.937304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757319.937307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757319.937313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757319.937318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757319.937322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757319.937325:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757319.937328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757319.937348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757320.129282:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757320.129289:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757320.129296:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757320.129298:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757320.129375:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757320.129380:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757320.129384:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757320.961307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757320.961317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757320.961320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757320.961326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757320.961331:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757320.961337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757320.961340:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757320.961343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757320.961347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757321.153363:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757321.153372:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757321.153382:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757321.153388:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757321.153393:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757321.153398:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757321.153403:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757321.153407:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757321.153411:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757321.153415:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757321.985273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757321.985281:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757321.985283:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757321.985288:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757321.985293:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757321.985296:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757321.985299:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757321.985302:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757321.985305:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757322.177312:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757322.177319:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757322.177327:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757322.177329:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757322.177376:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757322.177379:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757322.177382:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757322.433305:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757322.433316:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757322.433319:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757322.433323:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757322.433327:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757322.433335:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10377 00000100:00080000:2.0:1787757322.433346:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757322.433349:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757322.433351:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757322.433356:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757322.433359:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757322.433362:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757322.433364:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757322.433367:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757322.433369:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757322.433373:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757322.433376:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757322.433379:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757322.433382:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757322.433385:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433388:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757322.433391:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757322.433405:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757322.433409:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433412:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757322.433416:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757322.433422:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433437:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757322.433451:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659636352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757322.433464:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757322.433470:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433493:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757322.433498:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757322.433521:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757322.433523:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757322.433525:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757322.433529:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757322.433533:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10377 00000100:00080000:2.0:1787757322.433540:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757322.433544:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757322.433546:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757322.433550:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757322.433553:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757322.433562:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757322.433570:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757322.433580:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757322.433586:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757322.433600:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757322.433608:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757322.433615:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757322.433624:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757322.433631:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433636:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757322.433643:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757322.433651:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433653:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757322.433661:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757322.433662:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433664:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757322.433666:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757322.433673:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433685:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757322.433695:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659636480/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757322.433704:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757322.433707:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433712:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757322.433715:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757322.433718:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757322.433720:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757322.433725:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757322.433732:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757322.433738:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10377 00000100:00080000:2.0:1787757322.433746:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757322.433753:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757322.433760:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757322.433770:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757322.433776:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757322.433783:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757322.433790:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757322.433795:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757322.433801:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757322.433810:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757322.433818:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757322.433825:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757322.433828:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757322.433831:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433833:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757322.433836:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757322.433839:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433842:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757322.433848:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757322.433853:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433856:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757322.433859:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757322.433866:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433878:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757322.433889:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659636608/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757322.433927:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757322.433931:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.433938:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757322.433943:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757322.433952:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757322.433956:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757322.433960:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757322.433965:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757322.433973:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10377 00000100:00080000:2.0:1787757322.433979:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757322.433986:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757322.433989:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757322.433995:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757322.434000:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757322.434004:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757322.434013:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757322.434019:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757322.434024:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757322.434031:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757322.434038:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757322.434043:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757322.434050:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757322.434055:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.434060:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757322.434064:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757322.434072:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.434075:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757322.434081:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757322.434086:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.434090:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757322.434094:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757322.434097:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.434104:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757322.434118:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659636736/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757322.434141:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757322.434143:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757322.434148:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757323.009280:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757323.009290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757323.009294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757323.009300:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757323.009305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757323.009311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757323.009315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757323.009319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757323.009323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757323.201277:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757323.201285:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757323.201289:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757323.201292:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757323.201297:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757323.201299:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757323.201301:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757323.201308:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757323.201317:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757323.201318:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757323.201372:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757323.201376:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757323.201379:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757324.033283:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757324.033290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757324.033291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757324.033295:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757324.033298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757324.033301:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757324.033303:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757324.033305:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757324.033307:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757324.225261:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757324.225267:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757324.225271:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757324.225272:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757324.225323:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757324.225326:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757324.225329:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757325.057273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757325.057281:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757325.057283:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757325.057288:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757325.057291:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757325.057295:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757325.057297:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757325.057299:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757325.057302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757325.249326:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757325.249335:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757325.249343:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757325.249346:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757326.081321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757326.081331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757326.081334:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757326.081341:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757326.081345:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757326.081350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757326.081353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757326.081357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757326.081361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757326.273352:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757326.273363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757326.273373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757326.273375:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757326.273402:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757326.273405:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757326.273409:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757327.105337:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757327.105345:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757327.105348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757327.105353:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757327.105357:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757327.105362:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757327.105364:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757327.105367:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757327.105371:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757327.297324:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757327.297333:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757327.297339:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757327.297341:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757327.297492:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757327.297500:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757327.297523:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757327.553263:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757327.553273:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757327.553276:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757327.553279:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757327.553284:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757327.553291:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10382 00000100:00080000:2.0:1787757327.553302:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757327.553305:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757327.553307:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757327.553312:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757327.553314:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757327.553317:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757327.553319:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757327.553321:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757327.553332:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757327.553337:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757327.553340:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757327.553343:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757327.553346:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757327.553349:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553351:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757327.553354:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757327.553364:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757327.553366:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553369:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757327.553372:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757327.553376:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553388:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757327.553397:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659636864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757327.553408:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757327.553428:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553442:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757327.553445:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757327.553448:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757327.553450:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757327.553452:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757327.553454:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757327.553457:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10382 00000100:00080000:2.0:1787757327.553461:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757327.553464:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757327.553466:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757327.553469:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757327.553470:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757327.553472:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757327.553475:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757327.553477:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757327.553478:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757327.553482:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757327.553488:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757327.553494:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757327.553503:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757327.553512:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553515:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757327.553523:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757327.553532:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553537:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757327.553544:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757327.553548:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553551:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757327.553554:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757327.553559:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553571:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757327.553582:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659636992/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757327.553591:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757327.553594:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553598:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757327.553605:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757327.553612:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757327.553615:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757327.553616:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757327.553619:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757327.553621:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10382 00000100:00080000:2.0:1787757327.553626:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757327.553627:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757327.553628:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757327.553630:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757327.553632:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757327.553634:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757327.553637:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757327.553639:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757327.553640:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757327.553643:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757327.553646:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757327.553647:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757327.553659:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757327.553662:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553668:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757327.553671:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757327.553676:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553678:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757327.553684:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757327.553687:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553691:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757327.553694:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757327.553697:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553705:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757327.553712:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659637120/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757327.553725:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757327.553729:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553736:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757327.553740:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757327.553748:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757327.553752:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757327.553757:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757327.553759:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757327.553762:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10382 00000100:00080000:2.0:1787757327.553765:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757327.553773:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757327.553777:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757327.553785:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757327.553791:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757327.553795:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757327.553804:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757327.553811:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757327.553823:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757327.553834:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757327.553842:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757327.553850:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757327.553856:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757327.553862:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553866:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757327.553868:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757327.553871:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553876:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757327.553887:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757327.553891:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553893:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757327.553894:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757327.553897:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553903:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757327.553912:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659637248/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757327.553935:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757327.553941:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757327.553950:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757328.129244:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757328.129251:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757328.129252:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757328.129256:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757328.129259:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757328.129262:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757328.129264:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757328.129266:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757328.129268:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757328.321274:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757328.321280:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757328.321284:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757328.321285:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757328.321322:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757328.321326:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757328.321329:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757329.153330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757329.153342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757329.153347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757329.153356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757329.153363:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757329.153369:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757329.153373:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757329.153378:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757329.153444:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757329.345336:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757329.345345:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757329.345353:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757329.345356:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757330.177271:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757330.177280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757330.177283:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757330.177290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757330.177294:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757330.177299:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757330.177302:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757330.177305:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757330.177308:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757330.369438:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757330.369449:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757330.369459:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757330.369461:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757330.369505:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757330.369510:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757330.369515:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787757330.625488:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787757330.625535:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787757330.625552:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757330.625567:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757330.625591:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757330.625593:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757330.625596:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757330.625605:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757330.625611:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757330.625614:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757330.625617:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757330.625620:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757330.625627:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757330.625632:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757330.625643:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757330.625647:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757330.625650:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757330.625657:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757330.625665:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757330.625673:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757330.625678:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757330.625684:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:1.0:1787757330.625951:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757327/real 1787757330] req@ffff8ffb47a38a80 x1874588659637120/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757330.625964:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757330.625966:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757330.625971:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757330.625973:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757330.625975:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.625978:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757330.625980:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757330.625986:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659637120/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757330.625995:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.625998:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757330.626002:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626004:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757330.626013:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626017:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757330.626032:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757330.626037:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757330.626041:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757330.626045:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757330.626050:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659637120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757330.626056:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757330.626062:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757327/real 1787757330] req@ffff8ffb47a3a680 x1874588659636864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757330.626074:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757330.626076:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757330.626081:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757330.626083:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757330.626085:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626088:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757330.626091:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757330.626096:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659636864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757330.626106:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626109:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757330.626112:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626115:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757330.626117:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626120:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757330.626123:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757330.626127:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757330.626132:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757330.626136:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757330.626141:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659636864:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757330.626148:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626151:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757330.626154:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757330.626158:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757330.626164:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757330.626168:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757330.626171:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757330.626176:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757330.626178:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757330.626215:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757330.626218:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626220:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757330.626222:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757330.626225:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757330.626228:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757330.626231:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757330.626232:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757330.626235:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757330.626236:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757330.626239:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757330.626241:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626244:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757330.626246:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757330.626247:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757330.626251:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757330.626253:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757330.626254:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757330.626257:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757330.626259:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757330.626261:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757330.626263:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626265:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757330.626267:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757330.626269:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757330.626272:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757330.626273:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757330.626275:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757330.626278:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757330.626279:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757330.626283:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757330.626285:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626289:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757330.626299:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757330.626302:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757330.626305:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757330.626307:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757331.201242:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757331.201247:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757331.201249:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757331.201253:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757331.201256:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757331.201259:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757331.201261:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757331.201262:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757331.201264:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757331.393273:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757331.393278:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757331.393283:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757331.393285:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757331.393288:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757331.393290:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757331.393292:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757331.393294:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757331.393295:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757331.393297:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757331.393409:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757331.393412:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757331.393415:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757332.225363:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757332.225373:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757332.225376:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757332.225381:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757332.225385:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757332.225389:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757332.225391:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757332.225394:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757332.225397:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757332.417379:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757332.417389:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757332.417398:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757332.417401:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757332.417507:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757332.417513:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757332.417517:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757332.673391:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757332.673403:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757332.673407:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757332.673411:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757332.673417:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757332.673426:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10387 00000100:00080000:3.0:1787757332.673439:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757332.673443:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757332.673446:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757332.673451:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757332.673454:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757332.673459:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757332.673462:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757332.673465:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757332.673467:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757332.673473:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757332.673476:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757332.673480:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757332.673485:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757332.673489:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.673492:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757332.673495:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757332.673506:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757332.673509:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.673513:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757332.673516:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757332.673522:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.673535:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757332.673548:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659637376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757332.673561:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757332.673567:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.673589:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757332.673594:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757332.673598:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757332.673600:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757332.673603:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757332.673607:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757332.673610:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10387 00000100:00080000:3.0:1787757332.673614:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757332.673617:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757332.673619:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757332.673622:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757332.673624:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757332.673627:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757332.673631:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757332.673636:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757332.673639:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757332.673647:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757332.673653:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757332.673657:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757332.673666:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757332.673671:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.673676:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757332.673677:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787757332.673686:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757332.673689:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757332.673697:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:3.0:1787757332.673699:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.673705:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757332.673708:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.673714:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757332.673721:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757332.673725:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757332.673729:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.673745:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757332.673748:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757332.673757:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757332.673768:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659637376:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757332.673770:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757332.673779:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.673783:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757332.673794:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757332.673796:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757332.673800:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757332.673809:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787757332.673822:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659637504/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757332.673833:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.673854:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757332.673860:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.673868:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757332.673870:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.673878:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00080000:3.0:1787757332.673881:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757332.673894:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757332.673895:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757332.673903:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757332.673907:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757332.673914:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757332.673921:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10387 00000100:00080000:3.0:1787757332.673931:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757332.673937:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757332.673940:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757332.673946:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757332.673948:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757332.673952:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757332.673957:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757332.673962:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757332.673966:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757332.673974:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757332.673979:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757332.673984:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000800:00000100:1.0:1787757332.673993:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757332.673994:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757332.674003:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.674008:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757332.674014:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757332.674018:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00000001:3.0:1787757332.674022:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674029:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757332.674031:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674037:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757332.674044:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757332.674046:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757332.674052:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.674060:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757332.674061:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757332.674064:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757332.674073:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674075:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305060545 00000100:00000001:3.0:1787757332.674087:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674096:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757332.674111:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305060545 00000100:00100000:3.0:1787757332.674117:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659637632/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757332.674120:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757332.674133:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757332.674145:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.674148:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757332.674152:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757332.674159:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.674170:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757332.674173:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757332/real 1787757332] req@ffff8ffb45c30e00 x1874588659637376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757332.674178:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757332.674234:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757332.674238:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757332.674241:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757332.674251:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757332.674263:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10387 00000100:00080000:3.0:1787757332.674277:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000400:00000001:1.0:1787757332.674278:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757332.674284:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:1.0:1787757332.674287:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757332.674292:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757332.674300:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757332.674302:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000020:00000001:3.0:1787757332.674305:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757332.674313:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757332.674314:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000020:00000001:3.0:1787757332.674347:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757332.674351:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757332.674355:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757332.674364:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757332.674365:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757332.674373:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757332.674380:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757332.674392:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757332.674394:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659637376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757332.674399:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757332.674410:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757332.674421:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.674430:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757332.674434:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:3.0:1787757332.674438:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757332.674444:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00000001:3.0:1787757332.674449:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.674458:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674460:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.674467:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757332.674470:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757332.674472:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757332.674478:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674482:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757332.674488:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757332.674491:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787757332.674495:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757332.674502:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757332.674506:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757332.674517:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757332.674520:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757332.674533:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757332.674540:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659637760/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757332.674550:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659637376:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757332.674570:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.674575:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757332.674578:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757332.674582:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757332.674590:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757332.674594:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757332.674598:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757332.674605:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757332.674609:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757332.674614:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757332.674617:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.674622:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.674633:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757332.674638:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674643:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757332.674646:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757332.674649:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757332.674652:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757332.674655:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757332.674660:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757332.674663:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757332.674668:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757332.674676:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659637504:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757332.674683:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757332.674687:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757332.674690:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757332.674694:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757332.674700:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674713:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757332.674716:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757332.674724:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757332.674755:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757332.674766:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757332.674771:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757332.674774:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674777:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757332.674785:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674793:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305059854 00000100:00000001:1.0:1787757332.674801:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305060545 00000100:00000001:1.0:1787757332.674806:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305059854 00000100:00000001:1.0:1787757332.674817:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305060545 00000100:00000001:3.0:1787757332.674825:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757332.674826:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757332.674835:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757332.674837:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.674842:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757332.674852:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757332.674860:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757332/real 1787757332] req@ffff8ffb45c33b80 x1874588659637504/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757332.674882:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757332.674886:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757332.674894:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757332.674899:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757332.674903:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.674907:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757332.674909:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757332.674917:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659637504/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757332.674932:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.674937:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757332.674942:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.674945:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757332.674948:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.674952:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757332.674974:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757332.674980:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757332.674988:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757332.674992:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757332.674999:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659637504:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757332.675008:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757332.675011:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757332.675014:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757332.675019:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757332.675023:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757332.675027:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757332.675033:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659637632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757332.675051:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757332.675055:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757332.675058:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757332.675062:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757332.675069:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675080:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757332.675083:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757332.675091:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757332.675118:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757332.675127:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757332.675132:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757332.675135:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675139:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757332.675144:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675151:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305059851 00000100:00000001:1.0:1787757332.675168:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305060545 00000100:00000001:1.0:1787757332.675172:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305059851 00000100:00000001:1.0:1787757332.675230:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305060545 00000100:00000001:1.0:1787757332.675237:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757332.675242:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675245:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757332.675256:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757332/real 1787757332] req@ffff8ffb45c30a80 x1874588659637632/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757332.675274:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757332.675277:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757332.675283:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757332.675286:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757332.675289:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675292:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757332.675295:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757332.675302:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659637632/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757332.675315:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675345:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757332.675351:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675354:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757332.675356:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675360:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757332.675379:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757332.675384:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757332.675390:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757332.675395:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757332.675400:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659637632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757332.675408:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675412:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757332.675415:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757332.675418:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757332.675424:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757332.675427:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757332.675430:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757332.675434:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757332.675437:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757332.675441:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757332.675444:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675448:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757332.675451:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757332.675453:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757332.675457:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757332.675459:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757332.675477:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757332.675480:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757332.675482:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757332.675485:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757332.675487:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675491:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675498:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757332.675501:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675505:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757332.675508:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757332.675511:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757332.675513:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757332.675515:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757332.675520:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757332.675523:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757332.675527:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757332.675534:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659637760:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757332.675540:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757332.675543:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757332.675546:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757332.675549:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757332.675555:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675565:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757332.675568:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:1.0:1787757332.675575:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757332.675600:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757332.675610:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757332.675613:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757332.675615:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675618:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757332.675621:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675626:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305059852 00000100:00000001:1.0:1787757332.675630:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305060545 00000100:00000001:1.0:1787757332.675633:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305059852 00000100:00000001:1.0:1787757332.675636:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305060545 00000100:00000001:1.0:1787757332.675638:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757332.675641:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675643:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757332.675648:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757332/real 1787757332] req@ffff8ffb45c31c00 x1874588659637760/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757332.675657:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757332.675659:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757332.675661:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757332.675663:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757332.675665:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675667:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757332.675669:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757332.675673:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659637760/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757332.675682:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675685:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757332.675688:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675689:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757332.675691:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675693:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757332.675706:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757332.675709:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757332.675712:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757332.675714:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757332.675718:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659637760:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757332.675722:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675725:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757332.675727:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757332.675729:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757332.675731:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757332.675733:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757332.675735:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757332.675738:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757332.675739:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757332.675741:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757332.675743:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675745:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757332.675749:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757332.675751:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757332.675754:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757332.675756:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757333.249355:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757333.249371:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757333.249375:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757333.249383:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757333.249388:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757333.249394:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757333.249398:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757333.249402:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757333.249407:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757333.441329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757333.441340:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757333.441343:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757333.441347:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757333.441352:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757333.441355:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757333.441358:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757333.441360:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757333.441364:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757333.441365:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757333.441406:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757333.441411:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757333.441414:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757334.273392:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757334.273413:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757334.273416:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757334.273427:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757334.273432:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757334.273437:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757334.273440:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757334.273444:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757334.273447:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757334.465353:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757334.465364:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757334.465373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757334.465376:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757334.465412:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757334.465416:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757334.465420:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757335.297384:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757335.297399:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757335.297404:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757335.297414:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757335.297421:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757335.297427:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757335.297432:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757335.297437:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757335.297442:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757335.489370:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757335.489379:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757335.489386:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757335.489387:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757335.489481:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757335.489489:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757335.489494:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757336.321377:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757336.321390:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757336.321395:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757336.321404:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757336.321412:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757336.321418:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757336.321423:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757336.321429:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757336.321437:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757336.513358:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757336.513370:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757336.513380:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757336.513383:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757336.513482:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757336.513488:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757336.513494:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757337.345330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757337.345338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757337.345341:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757337.345347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757337.345351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757337.345355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757337.345358:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757337.345361:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757337.345365:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757337.537363:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757337.537373:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757337.537382:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757337.537384:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757337.537447:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757337.537452:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757337.537457:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757337.793390:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757337.793411:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757337.793414:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757337.793419:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757337.793426:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757337.793437:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10392 00000100:00080000:1.0:1787757337.793452:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757337.793457:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757337.793461:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757337.793467:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757337.793471:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757337.793476:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757337.793481:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757337.793484:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757337.793487:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757337.793495:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757337.793499:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757337.793503:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757337.793508:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757337.793512:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.793515:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757337.793519:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757337.793532:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757337.793535:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.793539:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757337.793543:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757337.793548:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.793563:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757337.793577:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659637888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757337.793593:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757337.793599:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.793621:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757337.793627:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757337.793632:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757337.793635:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757337.793639:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757337.793645:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757337.793645:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757337.793654:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:1.0:1787757337.793655:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10392 00000100:00000001:2.0:1787757337.793658:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757337.793663:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:1.0:1787757337.793667:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757337.793667:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.793675:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787757337.793677:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757337.793680:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757337.793682:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787757337.793687:0:87148:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:2.0:1787757337.793689:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757337.793692:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757337.793698:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00100000:2.0:1787757337.793699:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659637888:192.168.201.118@tcp:250:kworker.0 00000020:00000001:1.0:1787757337.793705:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757337.793708:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787757337.793712:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787757337.793715:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757337.793716:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757337.793719:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757337.793723:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:1.0:1787757337.793727:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787757337.793731:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.793734:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757337.793740:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757337.793747:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757337.793749:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787757337.793751:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.793753:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:1.0:1787757337.793758:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757337.793760:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00100000:1.0:1787757337.793768:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757337.793775:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.793779:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757337.793799:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000800:00000100:2.0:1787757337.793801:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:1.0:1787757337.793804:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.793810:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757337.793813:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757337.793816:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787757337.793818:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.793822:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757337.793826:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.793829:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757337.793830:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.793835:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.793841:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305065665 00000100:00100000:1.0:1787757337.793846:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659638016/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757337.793866:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757337.793868:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757337.793871:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.793876:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305065665 00000100:00000001:1.0:1787757337.793882:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.793883:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787757337.793891:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757337.793892:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757337.793895:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.793898:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757337.793905:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757337/real 1787757337] req@ffff8ffb4a535c00 x1874588659637888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757337.793918:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757337.793923:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000400:00000001:2.0:1787757337.793927:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:1.0:1787757337.793929:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000400:00000001:2.0:1787757337.793931:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757337.793938:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757337.793940:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757337.793943:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757337.793947:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757337.793949:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10392 00000100:00000001:2.0:1787757337.793951:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757337.793955:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757337.793960:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757337.793963:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659637888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757337.793967:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757337.793972:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757337.793981:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757337.793981:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787757337.793986:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757337.793988:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:1.0:1787757337.793991:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757337.793993:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.793999:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:1.0:1787757337.794001:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757337.794002:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757337.794010:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757337.794011:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757337.794016:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757337.794018:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757337.794020:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757337.794031:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1787757337.794033:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757337.794035:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1787757337.794044:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757337.794045:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659637888:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757337.794053:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757337.794056:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794059:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757337.794061:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757337.794062:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757337.794064:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757337.794069:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:1.0:1787757337.794070:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757337.794073:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:1.0:1787757337.794076:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000020:00000001:2.0:1787757337.794077:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000008:00100000:1.0:1787757337.794083:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787757337.794085:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757337.794088:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757337.794092:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000008:00000001:1.0:1787757337.794093:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794095:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794101:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757337.794103:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794111:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:1.0:1787757337.794114:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757337.794115:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757337.794120:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794120:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757337.794124:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757337.794128:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757337.794129:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757337.794134:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787757337.794135:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757337.794136:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757337.794141:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757337.794142:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757337.794146:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757337.794150:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757337.794153:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638016:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757337.794154:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794158:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757337.794162:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757337.794164:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757337.794168:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:1.0:1787757337.794170:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659638144/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757337.794173:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794215:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757337.794217:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787757337.794222:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757337.794258:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757337.794266:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757337.794267:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757337.794272:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757337.794274:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794276:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794281:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757337.794285:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794287:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757337.794293:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757337.794297:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305064974 00000100:00080000:1.0:1787757337.794303:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757337.794306:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305065665 00000100:00000001:1.0:1787757337.794308:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757337.794313:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757337.794315:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305064974 00000100:00000001:1.0:1787757337.794321:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757337.794323:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305065665 00000100:00080000:1.0:1787757337.794332:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10392 00000100:00000001:2.0:1787757337.794332:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757337.794340:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757337.794343:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757337.794346:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757337.794350:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757337.794354:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787757337.794359:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757337/real 1787757337] req@ffff8ffb4a534a80 x1874588659638016/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:1.0:1787757337.794360:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757337.794365:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757337.794370:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757337.794379:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757337.794387:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:2.0:1787757337.794391:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787757337.794392:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:2.0:1787757337.794397:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:1.0:1787757337.794405:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757337.794405:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757337.794411:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757337.794413:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794416:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757337.794419:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757337.794422:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757337.794426:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757337.794428:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:2.0:1787757337.794433:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659638016/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787757337.794437:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.794446:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757337.794450:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:1.0:1787757337.794452:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757337.794454:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757337.794460:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:1.0:1787757337.794464:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794465:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757337.794468:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757337.794472:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794473:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:1.0:1787757337.794484:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757337.794487:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.794490:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757337.794491:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:1.0:1787757337.794493:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00080000:2.0:1787757337.794496:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 02000000:00000001:1.0:1787757337.794499:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757337.794502:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757337.794507:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757337.794511:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638016:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757337.794512:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794517:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794521:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757337.794524:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00100000:1.0:1787757337.794528:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659638272/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757337.794528:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757337.794532:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757337.794535:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757337.794539:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757337.794543:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757337.794546:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757337.794550:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757337.794553:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757337.794554:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794559:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757337.794562:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794565:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757337.794570:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757337.794572:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794573:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757337.794576:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757337.794581:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757337.794583:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757337.794585:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757337.794588:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757337.794590:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757337.794593:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757337.794596:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757337.794600:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757337.794603:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757337.794604:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757337.794606:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757337.794610:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794616:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757337.794618:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787757337.794622:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757337.794635:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757337.794639:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757337.794641:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757337.794643:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794644:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757337.794647:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794650:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305064971 00000100:00000001:2.0:1787757337.794653:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305065665 00000100:00000001:2.0:1787757337.794655:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305064971 00000100:00000001:2.0:1787757337.794657:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305065665 00000100:00000001:2.0:1787757337.794659:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757337.794661:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794662:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757337.794666:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757337/real 1787757337] req@ffff8ffb4a535c00 x1874588659638144/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757337.794673:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757337.794674:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757337.794676:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757337.794677:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757337.794679:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794680:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757337.794682:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757337.794685:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659638144/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757337.794692:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794694:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757337.794695:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794697:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757337.794698:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794699:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757337.794709:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757337.794711:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757337.794714:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757337.794715:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757337.794718:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757337.794721:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757337.794722:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757337.794724:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757337.794726:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757337.794728:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757337.794729:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757337.794732:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638272:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757337.794735:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757337.794736:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757337.794737:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757337.794739:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757337.794741:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794745:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757337.794746:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757337.794749:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757337.794757:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757337.794761:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757337.794763:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757337.794764:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794766:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757337.794768:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794770:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305064971 00000100:00000001:2.0:1787757337.794773:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305065665 00000100:00000001:2.0:1787757337.794775:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305064971 00000100:00000001:2.0:1787757337.794777:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305065665 00000100:00000001:2.0:1787757337.794779:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757337.794781:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794782:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757337.794785:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757337/real 1787757337] req@ffff8ffb4a534700 x1874588659638272/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757337.794791:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757337.794792:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757337.794794:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757337.794795:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757337.794796:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794798:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757337.794799:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757337.794802:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659638272/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757337.794808:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794810:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757337.794812:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794814:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757337.794815:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794816:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757337.794826:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757337.794828:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757337.794831:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757337.794833:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757337.794836:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638272:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757337.794839:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794841:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757337.794842:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757337.794843:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757337.794845:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757337.794846:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757337.794847:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757337.794848:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757337.794849:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757337.794851:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757337.794852:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794854:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757337.794855:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757337.794856:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757337.794857:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757337.794858:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757337.794859:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757337.794861:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757337.794862:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757337.794863:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757337.794864:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794866:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757337.794868:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757337.794869:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757337.794871:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757337.794872:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757338.369289:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757338.369297:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757338.369300:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757338.369306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757338.369308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757338.369312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757338.369314:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757338.369316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757338.369318:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757338.561326:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757338.561368:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757338.561377:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757338.561380:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757338.561412:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757338.561415:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757338.561419:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757339.393440:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757339.393454:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757339.393459:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757339.393471:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757339.393479:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757339.393486:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757339.393491:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757339.393497:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757339.393503:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757339.585386:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757339.585396:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757339.585404:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757339.585406:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757339.585494:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757339.585502:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757339.585508:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757340.417284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757340.417294:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757340.417296:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757340.417302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757340.417307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757340.417311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757340.417314:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757340.417317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757340.417320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757340.609391:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757340.609402:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757340.609410:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757340.609413:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757340.609575:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757340.609583:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757340.609590:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757341.441340:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757341.441354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757341.441357:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757341.441365:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757341.441371:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757341.441376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757341.441380:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757341.441383:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757341.441387:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757341.633373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757341.633392:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757341.633402:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757341.633407:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757341.633411:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757341.633416:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757341.633421:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757341.633425:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757341.633428:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757341.633431:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757341.633519:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757341.633526:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757341.633532:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757342.465285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757342.465295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757342.465297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757342.465304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757342.465309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757342.465313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757342.465316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757342.465319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757342.465322:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757342.657305:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757342.657320:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757342.657334:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757342.657336:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757342.657369:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757342.657373:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757342.657377:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757342.913348:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757342.913364:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757342.913369:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757342.913375:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757342.913383:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757342.913395:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10397 00000100:00080000:3.0:1787757342.913425:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757342.913432:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757342.913435:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757342.913443:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757342.913449:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757342.913456:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757342.913461:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757342.913465:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757342.913467:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757342.913474:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757342.913478:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757342.913483:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757342.913490:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757342.913495:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.913498:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757342.913501:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757342.913518:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757342.913522:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.913529:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757342.913534:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757342.913541:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.913561:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757342.913576:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659638400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757342.913600:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757342.913607:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.913637:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757342.913644:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757342.913651:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757342.913656:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757342.913660:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757342.913668:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757342.913677:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10397 00000100:00080000:3.0:1787757342.913688:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757342.913693:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757342.913696:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757342.913701:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757342.913703:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757342.913708:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757342.913712:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757342.913716:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757342.913717:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757342.913719:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757342.913729:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757342.913731:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757342.913736:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.913739:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757342.913741:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757342.913749:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757342.913755:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757342.913755:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757342.913767:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.913768:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757342.913778:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757342.913778:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757342.913786:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757342.913790:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757342.913798:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659638400:192.168.201.118@tcp:250:kworker.0 00000008:00000001:3.0:1787757342.913802:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.913811:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757342.913813:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.913821:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757342.913826:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.913830:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757342.913838:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.913867:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757342.913871:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757342.913879:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757342.913959:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757342.913973:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757342.913978:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.913981:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757342.913982:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757342.913990:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757342.913994:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757342.913995:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914003:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914009:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757342.914011:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914019:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757342.914027:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659638528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757342.914031:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305070785 00000100:00000001:0.0:1787757342.914051:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757342.914062:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757342.914066:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305070785 00000100:00000001:3.0:1787757342.914069:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914079:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757342.914083:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914087:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757342.914096:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757342.914096:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757342.914103:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757342.914114:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757342/real 1787757342] req@ffff8ffb45c30000 x1874588659638400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757342.914138:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757342.914168:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757342.914177:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757342.914237:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757342.914243:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914248:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757342.914251:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757342.914262:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659638400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757342.914284:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914289:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757342.914294:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914297:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757342.914299:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914303:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757342.914306:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757342.914311:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757342.914318:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757342.914323:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757342.914329:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659638400:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757342.914337:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914340:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757342.914342:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757342.914345:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757342.914352:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757342.914355:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757342.914358:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757342.914365:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757342.914368:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757342.914372:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757342.914374:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914377:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914391:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757342.914396:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914404:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757342.914408:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757342.914412:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757342.914415:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757342.914419:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.914425:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757342.914429:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757342.914434:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757342.914437:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757342.914443:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757342.914451:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10397 02000000:00000001:0.0:1787757342.914452:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757342.914459:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787757342.914462:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.914466:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757342.914471:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757342.914476:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757342.914481:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:0.0:1787757342.914483:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638528:192.168.201.118@tcp:38:kworker.0 00000020:00000001:3.0:1787757342.914486:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757342.914495:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757342.914498:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757342.914501:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757342.914505:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757342.914506:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757342.914511:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.914515:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757342.914519:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914531:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757342.914534:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757342.914542:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757342.914570:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757342.914582:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757342.914585:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757342.914589:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757342.914593:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914596:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757342.914598:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757342.914607:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757342.914607:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757342.914615:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.914619:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757342.914621:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757342.914624:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757342.914632:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.914635:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914637:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305070094 02000000:00000001:3.0:1787757342.914643:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757342.914648:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305070785 02000000:00000001:3.0:1787757342.914649:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.914655:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757342.914660:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757342.914662:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305070094 02000000:00000001:3.0:1787757342.914665:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914670:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305070785 00000100:00000001:3.0:1787757342.914679:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914680:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757342.914691:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757342.914694:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659638656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757342.914700:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757342.914714:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:0.0:1787757342.914716:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757342/real 1787757342] req@ffff8ffb45c30700 x1874588659638528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757342.914722:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.914732:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757342.914741:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000400:00000001:0.0:1787757342.914745:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:3.0:1787757342.914748:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000400:00000001:0.0:1787757342.914750:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757342.914757:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757342.914761:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:3.0:1787757342.914764:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757342.914768:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757342.914773:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914778:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757342.914778:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757342.914786:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757342.914793:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10397 00000100:00100000:0.0:1787757342.914797:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659638528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757342.914808:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757342.914818:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757342.914824:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757342.914833:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757342.914836:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757342.914840:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787757342.914843:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757342.914848:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757342.914850:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757342.914856:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:3.0:1787757342.914860:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757342.914863:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757342.914870:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757342.914873:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757342.914874:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757342.914885:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757342.914890:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757342.914893:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757342.914898:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757342.914902:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.914905:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757342.914909:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757342.914915:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914916:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757342.914918:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757342.914931:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 02000000:00000001:3.0:1787757342.914933:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757342.914941:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757342.914947:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757342.914949:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757342.914953:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1787757342.914957:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:0.0:1787757342.914962:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638528:192.168.201.118@tcp:38:kworker.0 02000000:00000001:3.0:1787757342.914966:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.914972:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.914978:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757342.914983:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757342.914983:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.914989:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757342.914996:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757342.914999:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757342.915004:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00100000:3.0:1787757342.915007:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659638784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757342.915009:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757342.915014:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757342.915020:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757342.915024:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915032:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757342.915039:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757342.915041:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757342.915046:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915048:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757342.915053:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915055:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757342.915063:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757342.915065:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757342.915069:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757342.915071:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.915075:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757342.915078:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757342.915080:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757342.915087:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757342.915093:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757342.915096:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757342.915098:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.915101:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757342.915106:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915114:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757342.915117:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757342.915124:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757342.915175:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757342.915227:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757342.915234:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757342.915238:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915242:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757342.915248:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915254:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305070092 00000100:00000001:0.0:1787757342.915258:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305070785 00000100:00000001:0.0:1787757342.915261:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305070092 00000100:00000001:0.0:1787757342.915264:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305070785 00000100:00000001:0.0:1787757342.915267:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757342.915270:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915272:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757342.915279:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757342/real 1787757342] req@ffff8ffb45c32a00 x1874588659638656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757342.915289:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757342.915291:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757342.915295:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757342.915297:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757342.915299:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915301:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757342.915304:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757342.915309:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659638656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757342.915326:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915329:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757342.915333:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915335:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757342.915336:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915338:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757342.915357:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757342.915360:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757342.915364:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757342.915367:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757342.915370:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757342.915376:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757342.915378:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757342.915379:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.915383:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757342.915387:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757342.915391:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757342.915397:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638784:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757342.915404:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757342.915407:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757342.915410:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757342.915413:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757342.915420:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915431:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757342.915434:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787757342.915442:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757342.915468:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757342.915477:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757342.915482:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757342.915485:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915490:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757342.915495:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915499:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305070092 00000100:00000001:0.0:1787757342.915503:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305070785 00000100:00000001:0.0:1787757342.915507:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305070092 00000100:00000001:0.0:1787757342.915509:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305070785 00000100:00000001:0.0:1787757342.915512:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757342.915515:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915517:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757342.915523:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757342/real 1787757342] req@ffff8ffb45c33800 x1874588659638784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757342.915533:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757342.915534:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757342.915537:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757342.915539:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757342.915541:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915544:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757342.915546:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757342.915551:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659638784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757342.915559:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915562:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757342.915565:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915567:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757342.915568:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915570:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757342.915591:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757342.915596:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757342.915602:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757342.915607:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757342.915614:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659638784:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757342.915621:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915625:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757342.915627:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757342.915630:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757342.915635:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757342.915638:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757342.915641:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757342.915645:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757342.915648:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757342.915652:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757342.915655:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915659:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757342.915661:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757342.915663:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757342.915668:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757342.915671:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757342.915673:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757342.915677:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757342.915680:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757342.915683:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757342.915685:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915689:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757342.915695:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757342.915697:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757342.915700:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757342.915702:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757343.489303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757343.489312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757343.489315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757343.489320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757343.489324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757343.489329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757343.489332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757343.489334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757343.489337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757343.681304:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757343.681313:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757343.681316:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757343.681335:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757343.681339:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757343.681341:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757343.681343:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757343.681345:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757343.681357:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757343.681360:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757343.681399:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757343.681404:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757343.681407:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757344.513346:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757344.513358:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757344.513360:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757344.513368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757344.513374:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757344.513391:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757344.513395:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757344.513399:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757344.513403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757344.705333:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757344.705342:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757344.705349:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757344.705352:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757344.705432:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757344.705457:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757344.705462:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757345.537376:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757345.537387:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757345.537391:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757345.537398:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757345.537404:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757345.537410:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757345.537414:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757345.537418:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757345.537422:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757345.729354:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757345.729364:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757345.729373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757345.729374:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757345.729443:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757345.729448:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757345.729453:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757346.561293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757346.561303:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757346.561305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757346.561310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757346.561314:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757346.561318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757346.561321:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757346.561323:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757346.561326:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757346.753326:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757346.753334:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757346.753340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757346.753341:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757346.753362:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757346.753365:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757346.753368:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757347.585376:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757347.585389:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757347.585393:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757347.585401:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757347.585407:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757347.585415:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757347.585420:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757347.585427:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757347.585433:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757347.777291:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757347.777301:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757347.777309:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757347.777310:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757347.777393:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757347.777398:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757347.777402:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757348.033302:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757348.033314:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757348.033318:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757348.033322:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757348.033328:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757348.033338:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10402 00000100:00080000:3.0:1787757348.033352:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757348.033356:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757348.033359:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757348.033364:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757348.033367:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757348.033372:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757348.033375:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757348.033378:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757348.033380:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757348.033385:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757348.033388:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757348.033391:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757348.033397:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757348.033401:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033403:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757348.033406:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757348.033418:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757348.033421:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033425:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757348.033428:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757348.033433:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033447:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757348.033459:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659638912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757348.033472:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757348.033478:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033499:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757348.033504:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757348.033509:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757348.033511:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757348.033513:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757348.033516:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757348.033519:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10402 00000100:00080000:3.0:1787757348.033523:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757348.033525:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757348.033527:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757348.033529:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757348.033531:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757348.033534:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757348.033536:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757348.033539:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757348.033541:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757348.033545:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757348.033548:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757348.033552:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757348.033558:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757348.033562:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033575:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757348.033582:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757348.033586:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.033590:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757348.033590:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.033601:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757348.033602:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757348.033609:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.033610:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757348.033613:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757348.033616:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757348.033618:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757348.033621:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757348.033623:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757348.033636:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757348.033637:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757348.033643:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00100000:3.0:1787757348.033651:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659639040/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757348.033652:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757348.033675:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659638912:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757348.033692:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757348.033700:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757348.033701:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757348.033708:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033712:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757348.033714:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757348.033726:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033726:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757348.033739:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757348.033747:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757348.033750:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757348.033753:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757348.033755:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757348.033759:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787757348.033766:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757348.033768:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757348.033780:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10402 00000100:00080000:3.0:1787757348.033791:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757348.033796:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757348.033800:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757348.033806:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757348.033810:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757348.033814:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757348.033821:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000800:00000100:0.0:1787757348.033823:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757348.033831:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757348.033838:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757348.033843:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757348.033852:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757348.033853:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757348.033860:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033863:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.033865:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757348.033872:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757348.033876:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757348.033881:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757348.033885:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305075905 02000000:00000001:3.0:1787757348.033892:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033899:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757348.033902:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00100000:3.0:1787757348.033905:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757348.033911:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305075905 00000008:00000001:3.0:1787757348.033914:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033922:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.033923:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757348.033931:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757348.033934:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757348.033935:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757348.033943:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.033944:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757348.033952:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:0.0:1787757348.033957:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757348/real 1787757348] req@ffff8ffb45c31500 x1874588659638912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757348.033959:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757348.033966:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.033981:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757348.033984:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757348.033990:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757348.033995:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757348.034001:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00100000:3.0:1787757348.034003:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659639168/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757348.034004:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034011:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757348.034016:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757348.034032:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659638912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757348.034037:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757348.034047:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034074:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034088:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757348.034089:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034094:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757348.034100:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757348.034101:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757348.034106:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034113:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00080000:3.0:1787757348.034117:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757348.034118:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757348.034126:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757348.034128:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757348.034134:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757348.034138:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757348.034146:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757348.034147:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00080000:3.0:1787757348.034158:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10402 00000100:00100000:0.0:1787757348.034159:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659638912:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757348.034175:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757348.034176:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757348.034178:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757348.034217:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757348.034221:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757348.034227:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757348.034230:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757348.034233:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757348.034238:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787757348.034242:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757348.034243:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757348.034251:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757348.034252:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757348.034257:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757348.034263:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757348.034264:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757348.034272:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757348.034278:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000020:00000001:3.0:1787757348.034281:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757348.034284:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757348.034292:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757348.034293:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757348.034297:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757348.034302:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757348.034305:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757348.034307:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757348.034313:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.034314:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757348.034317:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757348.034324:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757348.034334:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757348.034334:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1787757348.034346:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639040:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757348.034348:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757348.034361:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757348.034362:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757348.034370:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757348.034371:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757348.034375:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757348.034381:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757348.034384:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757348.034389:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757348.034395:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757348.034398:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757348.034406:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.034410:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034414:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757348.034421:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757348.034422:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 02000000:00000001:3.0:1787757348.034428:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.034434:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757348.034436:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757348.034438:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757348.034446:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.034459:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757348.034470:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659639296/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757348.034478:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757348.034494:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757348.034496:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757348.034504:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034505:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757348.034515:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757348.034517:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034521:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757348.034534:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034545:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305075214 00000100:00000001:0.0:1787757348.034552:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305075905 00000100:00000001:0.0:1787757348.034557:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305075214 00000100:00000001:0.0:1787757348.034560:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305075905 00000100:00000001:0.0:1787757348.034564:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757348.034567:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034570:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757348.034577:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757348/real 1787757348] req@ffff8ffb45c33480 x1874588659639040/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757348.034597:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757348.034599:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757348.034603:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757348.034606:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757348.034608:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034610:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757348.034612:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757348.034617:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659639040/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757348.034626:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034629:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757348.034632:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034634:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757348.034635:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034638:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757348.034653:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757348.034656:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757348.034660:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757348.034662:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757348.034666:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639040:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757348.034681:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757348.034684:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757348.034685:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757348.034689:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757348.034691:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757348.034694:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757348.034697:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757348.034701:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757348.034704:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757348.034706:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757348.034709:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757348.034713:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034720:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757348.034722:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787757348.034726:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757348.034743:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757348.034749:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757348.034751:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757348.034753:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034756:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757348.034759:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034763:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305075211 00000100:00000001:0.0:1787757348.034766:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305075905 00000100:00000001:0.0:1787757348.034769:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305075211 00000100:00000001:0.0:1787757348.034772:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305075905 00000100:00000001:0.0:1787757348.034775:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757348.034777:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034779:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757348.034784:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757348/real 1787757348] req@ffff8ffb45c31180 x1874588659639168/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757348.034793:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757348.034795:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757348.034798:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757348.034800:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757348.034801:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034804:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757348.034805:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757348.034810:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659639168/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757348.034818:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034820:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757348.034823:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034825:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757348.034826:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034828:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757348.034842:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757348.034845:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757348.034849:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757348.034851:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757348.034855:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757348.034860:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034862:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757348.034863:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757348.034865:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757348.034868:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757348.034870:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757348.034871:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757348.034874:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757348.034875:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757348.034878:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757348.034880:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034882:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757348.034883:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757348.034885:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757348.034888:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757348.034890:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757348.034891:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757348.034893:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757348.034895:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757348.034897:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757348.034899:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034901:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.034906:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757348.034908:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034910:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757348.034912:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757348.034914:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757348.034916:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757348.034918:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757348.034920:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757348.034923:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757348.034925:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757348.034929:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639296:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757348.034932:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757348.034934:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757348.034936:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757348.034938:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757348.034941:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034946:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757348.034948:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787757348.034952:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757348.034964:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757348.034969:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757348.034972:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757348.034974:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034976:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757348.034979:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034983:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305075211 00000100:00000001:0.0:1787757348.034985:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305075905 00000100:00000001:0.0:1787757348.034988:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305075211 00000100:00000001:0.0:1787757348.034991:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305075905 00000100:00000001:0.0:1787757348.034995:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757348.034997:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.034999:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757348.035004:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757348/real 1787757348] req@ffff8ffb45c31500 x1874588659639296/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757348.035014:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757348.035015:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757348.035018:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757348.035020:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757348.035022:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.035023:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757348.035025:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757348.035029:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659639296/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757348.035037:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.035040:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757348.035042:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.035044:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757348.035046:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.035078:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757348.035093:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757348.035097:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757348.035102:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757348.035105:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757348.035109:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639296:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757348.035114:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.035117:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757348.035118:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757348.035120:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757348.035123:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757348.035125:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757348.035128:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757348.035130:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757348.035132:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757348.035134:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757348.035136:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.035138:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757348.035142:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757348.035144:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757348.035146:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757348.035149:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757348.609415:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757348.609430:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757348.609436:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757348.609476:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757348.609484:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757348.609493:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757348.609498:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757348.609502:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757348.609507:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757348.801259:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757348.801265:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757348.801271:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757348.801273:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757348.801289:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757348.801295:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757348.801302:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757349.633293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757349.633302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757349.633304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757349.633309:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757349.633313:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757349.633316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757349.633319:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757349.633321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757349.633323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757349.825251:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757349.825257:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757349.825261:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757349.825263:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757349.825371:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757349.825376:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757349.825380:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757350.657364:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757350.657388:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757350.657392:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757350.657401:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757350.657407:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757350.657413:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757350.657417:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757350.657421:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757350.657424:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757350.849272:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757350.849279:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757350.849286:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757350.849288:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757350.849341:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757350.849344:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757350.849348:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757351.681291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757351.681301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757351.681304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757351.681311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757351.681316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757351.681321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757351.681324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757351.681328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757351.681344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757351.873336:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757351.873345:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757351.873354:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757351.873358:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757351.873362:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757351.873367:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757351.873372:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757351.873375:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757351.873377:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757351.873380:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757351.873491:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757351.873496:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757351.873500:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757352.705313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757352.705322:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757352.705325:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757352.705332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757352.705337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757352.705342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757352.705346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757352.705349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757352.705353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757352.897305:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757352.897314:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757352.897320:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757352.897322:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757352.897406:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757352.897414:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757352.897421:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757353.153353:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757353.153369:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757353.153373:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757353.153377:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757353.153384:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757353.153394:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10407 00000100:00080000:2.0:1787757353.153408:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757353.153412:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757353.153415:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757353.153420:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757353.153423:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757353.153427:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757353.153431:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757353.153434:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757353.153436:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757353.153442:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757353.153445:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757353.153449:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757353.153454:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757353.153458:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153461:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757353.153464:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757353.153479:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757353.153481:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153486:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757353.153489:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757353.153494:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153509:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757353.153522:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659639424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757353.153535:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757353.153541:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153564:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757353.153569:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757353.153573:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757353.153575:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757353.153578:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757353.153582:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757353.153585:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10407 00000100:00080000:2.0:1787757353.153590:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757353.153593:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757353.153596:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757353.153599:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757353.153602:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757353.153604:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757353.153607:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757353.153610:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757353.153612:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757353.153616:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787757353.153617:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757353.153623:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757353.153626:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757353.153633:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757353.153636:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757353.153638:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.153642:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757353.153644:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757353.153648:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:2.0:1787757353.153653:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757353.153654:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757353.153665:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.153668:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757353.153670:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.153675:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757353.153679:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757353.153683:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757353.153684:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153693:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757353.153696:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:0.0:1787757353.153702:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659639424:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757353.153703:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153714:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.153715:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757353.153725:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:2.0:1787757353.153729:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659639552/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757353.153730:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.153739:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757353.153750:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757353.153751:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153755:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153763:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757353.153767:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757353.153771:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757353.153774:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757353.153775:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787757353.153777:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757353.153781:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:2.0:1787757353.153790:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757353.153792:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787757353.153804:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10407 00000100:00080000:2.0:1787757353.153814:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757353.153819:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757353.153822:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757353.153827:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757353.153831:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757353.153836:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757353.153841:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000800:00000100:0.0:1787757353.153848:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757353.153851:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757353.153859:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757353.153867:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:2.0:1787757353.153871:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757353.153875:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757353.153883:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153883:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757353.153894:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757353.153895:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787757353.153904:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757353.153907:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757353.153910:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.153917:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305081025 00000100:00000001:2.0:1787757353.153917:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757353.153924:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757353.153930:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153933:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.153935:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757353.153942:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757353.153945:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305081025 02000000:00000001:2.0:1787757353.153947:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.153954:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757353.153956:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757353.153980:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757353.153989:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757353.153993:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.153996:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757353.154005:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757353/real 1787757353] req@ffff8ffb47a39500 x1874588659639424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757353.154020:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:2.0:1787757353.154022:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757353.154024:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757353.154033:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757353.154036:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757353.154038:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154040:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154047:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757353.154049:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757353.154054:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659639680/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757353.154063:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659639424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757353.154089:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757353.154097:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154098:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757353.154103:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154106:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757353.154110:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757353.154116:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757353.154119:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757353.154121:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757353.154123:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757353.154126:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757353.154129:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757353.154132:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757353.154137:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757353.154139:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757353.154147:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10407 00000100:00080000:0.0:1787757353.154149:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757353.154157:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757353.154160:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757353.154165:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757353.154168:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757353.154170:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757353.154178:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:0.0:1787757353.154180:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659639424:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757353.154246:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757353.154249:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757353.154251:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:2.0:1787757353.154255:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757353.154257:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757353.154263:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000020:00000001:2.0:1787757353.154266:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:0.0:1787757353.154271:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:2.0:1787757353.154274:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000020:00000001:0.0:1787757353.154277:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:2.0:1787757353.154281:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757353.154282:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757353.154289:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757353.154293:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00080000:2.0:1787757353.154298:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757353.154301:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757353.154307:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757353.154310:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154312:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757353.154323:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757353.154328:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:2.0:1787757353.154333:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757353.154334:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154344:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757353.154346:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154350:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757353.154355:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757353.154356:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757353.154364:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:2.0:1787757353.154367:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757353.154368:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.154374:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:2.0:1787757353.154380:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.154381:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757353.154387:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757353.154395:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757353.154403:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757353.154404:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.154415:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:0.0:1787757353.154416:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639552:192.168.201.118@tcp:38:kworker.0 02000000:00000001:2.0:1787757353.154424:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757353.154427:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757353.154433:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.154434:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757353.154438:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.154444:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757353.154449:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757353.154452:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154457:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757353.154462:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:0.0:1787757353.154469:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:2.0:1787757353.154472:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659639808/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757353.154493:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757353.154495:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757353.154501:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154503:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757353.154508:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154511:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757353.154518:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154521:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757353.154526:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154532:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305080334 00000100:00000001:0.0:1787757353.154536:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305081025 00000100:00000001:0.0:1787757353.154540:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305080334 00000100:00000001:0.0:1787757353.154545:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305081025 00000100:00000001:0.0:1787757353.154548:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757353.154551:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154554:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757353.154561:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757353/real 1787757353] req@ffff8ffb47a39880 x1874588659639552/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757353.154573:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757353.154575:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757353.154579:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757353.154581:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757353.154584:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154586:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757353.154588:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757353.154593:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659639552/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757353.154602:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154604:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757353.154607:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154609:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757353.154611:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154613:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757353.154627:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757353.154630:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757353.154634:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757353.154637:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757353.154641:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639552:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757353.154645:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757353.154647:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757353.154649:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.154652:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757353.154654:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757353.154657:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757353.154661:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639680:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757353.154665:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757353.154667:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757353.154669:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.154671:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757353.154675:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154680:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757353.154682:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757353.154686:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757353.154700:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757353.154705:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757353.154707:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757353.154709:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154712:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757353.154715:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154719:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305080331 00000100:00000001:0.0:1787757353.154722:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305081025 00000100:00000001:0.0:1787757353.154725:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305080331 00000100:00000001:0.0:1787757353.154728:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305081025 00000100:00000001:0.0:1787757353.154731:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757353.154733:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154735:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757353.154740:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757353/real 1787757353] req@ffff8ffb47a3a680 x1874588659639680/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757353.154749:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757353.154751:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757353.154753:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757353.154755:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757353.154757:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154759:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757353.154761:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757353.154766:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659639680/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757353.154774:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154777:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757353.154780:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154781:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757353.154783:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154785:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757353.154797:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757353.154800:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757353.154804:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757353.154807:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757353.154811:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639680:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757353.154816:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154818:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757353.154819:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757353.154821:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757353.154824:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757353.154825:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757353.154827:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757353.154829:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757353.154831:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757353.154833:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757353.154835:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154837:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757353.154839:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757353.154840:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757353.154843:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757353.154844:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757353.154846:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757353.154848:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757353.154849:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757353.154852:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757353.154853:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154855:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.154859:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757353.154861:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154863:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757353.154865:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757353.154866:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757353.154868:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757353.154869:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.154872:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757353.154874:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757353.154876:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757353.154880:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639808:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757353.154884:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757353.154886:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757353.154888:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757353.154890:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757353.154894:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154899:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757353.154900:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757353.154905:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757353.154917:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757353.154922:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757353.154924:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757353.154926:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154928:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757353.154931:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154935:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305080331 00000100:00000001:0.0:1787757353.154938:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305081025 00000100:00000001:0.0:1787757353.154941:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305080331 00000100:00000001:0.0:1787757353.154944:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305081025 00000100:00000001:0.0:1787757353.154946:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757353.154948:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.154951:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757353.154955:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757353/real 1787757353] req@ffff8ffb47a38a80 x1874588659639808/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757353.154992:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757353.154994:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757353.154997:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757353.154999:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757353.155001:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.155003:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757353.155005:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757353.155010:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659639808/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757353.155019:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.155022:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757353.155026:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.155028:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757353.155030:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.155032:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757353.155044:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757353.155048:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757353.155052:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757353.155054:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757353.155058:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659639808:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757353.155062:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.155064:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757353.155066:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757353.155068:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757353.155070:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757353.155071:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757353.155073:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757353.155075:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757353.155077:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757353.155081:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757353.155082:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.155085:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757353.155088:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757353.155090:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757353.155093:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757353.155095:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757353.729325:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757353.729334:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757353.729337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757353.729343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757353.729348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757353.729353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757353.729356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757353.729360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757353.729363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757353.921311:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757353.921322:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757353.921326:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757353.921330:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757353.921338:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757353.921342:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757353.921358:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757353.921362:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757353.921369:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757353.921372:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757353.921424:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757353.921433:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757353.921442:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757354.753324:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757354.753335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757354.753339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757354.753346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757354.753352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757354.753358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757354.753363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757354.753366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757354.753371:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757354.945343:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757354.945353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757354.945361:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757354.945363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757354.945438:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757354.945442:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757354.945446:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757355.777354:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757355.777364:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757355.777367:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757355.777374:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757355.777380:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757355.777385:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757355.777390:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757355.777393:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757355.777397:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757355.969429:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757355.969444:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757355.969457:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757355.969460:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757355.969564:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757355.969573:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757355.969579:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757356.801374:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757356.801381:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757356.801383:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757356.801387:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757356.801390:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757356.801405:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757356.801407:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757356.801410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757356.801412:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757356.993381:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757356.993390:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757356.993399:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757356.993400:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757356.993510:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757356.993517:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757356.993522:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757357.825365:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757357.825374:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757357.825388:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757357.825395:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757357.825399:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757357.825405:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757357.825408:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757357.825412:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757357.825417:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757358.017284:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757358.017291:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757358.017297:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757358.017299:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757358.017375:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757358.017379:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757358.017383:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757358.273391:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757358.273412:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757358.273418:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757358.273424:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757358.273431:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757358.273441:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10413 00000100:00080000:3.0:1787757358.273459:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757358.273464:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757358.273467:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757358.273473:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757358.273477:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757358.273482:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757358.273485:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757358.273488:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757358.273490:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757358.273496:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757358.273500:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757358.273504:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757358.273509:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757358.273514:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.273517:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757358.273520:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757358.273539:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757358.273542:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.273547:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757358.273551:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757358.273558:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.273578:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757358.273593:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659639936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757358.273608:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757358.273615:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.273642:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757358.273647:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757358.273655:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757358.273657:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757358.273659:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757358.273664:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757358.273667:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10413 00000100:00080000:3.0:1787757358.273671:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757358.273674:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757358.273676:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757358.273679:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757358.273682:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757358.273685:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757358.273688:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757358.273691:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757358.273694:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757358.273698:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757358.273701:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757358.273703:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757358.273708:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757358.273712:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757358.273719:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757358.273720:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.273728:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757358.273731:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:3.0:1787757358.273734:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757358.273739:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:3.0:1787757358.273741:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.273750:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.273750:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757358.273760:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757358.273764:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.273767:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757358.273768:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757358.273775:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757358.273778:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757358.273779:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.273786:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757358.273795:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659640064/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757358.273810:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757358.273829:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659639936:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757358.273851:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757358.273854:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757358.273863:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757358.273864:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.273892:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.273902:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757358.273904:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.273915:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757358.273916:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757358.273929:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757358.273931:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757358.273934:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757358.273940:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757358.273946:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10413 00000100:00080000:3.0:1787757358.273952:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757358.273955:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757358.273956:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757358.273964:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:3.0:1787757358.273965:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757358.273974:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757358.273979:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757358.273980:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787757358.273984:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757358.273990:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757358.273992:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757358.273994:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757358.273998:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757358.274001:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757358.274004:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757358.274008:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757358.274011:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.274014:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757358.274018:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757358.274021:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.274023:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757358.274031:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757358.274033:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.274037:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757358.274040:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757358.274043:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.274052:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757358.274062:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1787757358.274064:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659640192/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757358.274079:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757358.274082:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.274086:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757358.274090:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757358.274094:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757358.274096:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757358.274098:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757358.274102:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757358.274103:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757358.274112:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10413 00000100:00000001:0.0:1787757358.274116:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757358.274119:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757358.274123:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.274125:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757358.274131:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757358.274132:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757358.274139:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757358.274141:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757358.274144:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757358.274145:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757358.274155:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757358.274158:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305086145 00000100:00000001:3.0:1787757358.274160:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757358.274167:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757358.274173:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757358.274177:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757358.274179:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757358.274181:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787757358.274236:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305086146 00000100:00000001:0.0:1787757358.274246:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757358.274250:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757358.274252:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757358.274261:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274263:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.274270:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757358.274273:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:3.0:1787757358.274278:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757358.274287:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757358/real 1787757358] req@ffff8ffb45c33800 x1874588659639936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:3.0:1787757358.274291:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757358.274302:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757358.274318:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:0.0:1787757358.274320:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757358.274328:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757358.274328:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274338:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757358.274339:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757358.274344:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787757358.274347:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757358.274349:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274357:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757358.274358:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274362:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757358.274372:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659639936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757358.274374:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274390:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757358.274395:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659640320/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757358.274398:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757358.274407:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274410:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757358.274411:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274415:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757358.274418:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757358.274423:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757358.274435:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757358.274444:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757358.274447:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757358.274455:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757358.274456:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659639936:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757358.274460:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274467:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274473:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757358.274476:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757358.274480:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757358.274487:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757358.274489:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757358.274492:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757358.274499:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757358.274502:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757358.274506:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757358.274508:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274511:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274521:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757358.274525:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274528:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757358.274530:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757358.274532:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757358.274535:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757358.274537:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.274540:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757358.274543:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757358.274546:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757358.274550:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640064:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757358.274555:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757358.274558:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757358.274560:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.274562:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757358.274566:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274574:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757358.274576:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757358.274582:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757358.274599:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757358.274605:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757358.274608:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757358.274610:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274613:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757358.274617:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274623:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305085454 00000100:00000001:0.0:1787757358.274627:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305086145 00000100:00000001:0.0:1787757358.274630:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305085454 00000100:00000001:0.0:1787757358.274633:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305086146 00000100:00000001:0.0:1787757358.274636:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757358.274638:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274641:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757358.274646:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757358/real 1787757358] req@ffff8ffb45c32a00 x1874588659640064/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757358.274656:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757358.274659:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757358.274662:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757358.274665:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757358.274669:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274672:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757358.274675:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757358.274682:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659640064/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757358.274697:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274702:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757358.274707:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274710:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757358.274712:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.274716:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757358.274744:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757358.274750:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757358.274757:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757358.274762:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757358.274770:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640064:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757358.274778:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757358.274781:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757358.274783:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.274788:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757358.274792:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757358.274796:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757358.274802:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757358.274808:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757358.274812:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757358.274816:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.274819:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757358.274824:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274833:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757358.274835:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757358.274842:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757358.274895:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757358.274906:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757358.274911:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757358.274914:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274917:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757358.274923:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274930:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305085451 00000100:00000001:0.0:1787757358.274936:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305086145 00000100:00000001:0.0:1787757358.274941:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305085451 00000100:00000001:0.0:1787757358.274946:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305086146 00000100:00000001:0.0:1787757358.274950:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757358.274955:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.274958:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757358.274968:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757358/real 1787757358] req@ffff8ffb45c30700 x1874588659640192/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757358.274998:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757358.275001:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757358.275006:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757358.275009:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757358.275011:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275013:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757358.275015:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757358.275021:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659640192/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757358.275030:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275034:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757358.275037:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275039:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757358.275040:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275042:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757358.275064:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757358.275069:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757358.275075:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757358.275079:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757358.275086:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757358.275093:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757358.275096:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757358.275099:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.275104:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757358.275107:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757358.275111:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757358.275117:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640320:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757358.275124:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757358.275128:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757358.275132:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757358.275136:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757358.275142:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.275153:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757358.275156:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757358.275164:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757358.275233:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757358.275244:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757358.275249:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757358.275251:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.275254:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757358.275258:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.275263:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305085452 00000100:00000001:0.0:1787757358.275267:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305086145 00000100:00000001:0.0:1787757358.275270:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305085452 00000100:00000001:0.0:1787757358.275273:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305086146 00000100:00000001:0.0:1787757358.275276:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757358.275279:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.275281:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757358.275287:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757358/real 1787757358] req@ffff8ffb45c30000 x1874588659640320/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757358.275298:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757358.275300:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757358.275303:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757358.275305:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757358.275307:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275310:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757358.275312:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757358.275317:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659640320/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757358.275326:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275329:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757358.275333:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275335:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757358.275336:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275339:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757358.275359:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757358.275363:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757358.275367:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757358.275370:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757358.275374:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640320:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757358.275378:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275381:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757358.275382:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757358.275385:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757358.275388:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757358.275390:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757358.275392:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757358.275395:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757358.275397:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757358.275400:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757358.275402:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275404:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757358.275406:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757358.275408:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757358.275411:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757358.275412:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757358.275413:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757358.275415:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757358.275417:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757358.275419:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757358.275420:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275422:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757358.275423:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757358.275425:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757358.275428:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757358.275429:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757358.275431:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757358.275432:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757358.275435:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757358.275436:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757358.275438:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275440:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757358.275446:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757358.275448:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757358.275451:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757358.275453:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757358.849375:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757358.849386:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757358.849390:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757358.849399:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757358.849406:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757358.849413:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757358.849418:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757358.849422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757358.849426:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757359.041416:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757359.041426:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757359.041434:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757359.041436:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757359.041487:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757359.041491:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757359.041495:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757359.873320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757359.873329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757359.873332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757359.873339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757359.873344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757359.873349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757359.873352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757359.873355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757359.873359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757360.065367:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757360.065378:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757360.065386:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757360.065388:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757360.065506:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757360.065515:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757360.065522:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757360.897335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757360.897342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757360.897344:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757360.897348:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757360.897352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757360.897355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757360.897358:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757360.897360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757360.897363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757361.089376:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757361.089386:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757361.089393:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757361.089395:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757361.089504:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757361.089512:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757361.089518:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757361.921338:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757361.921351:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757361.921354:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757361.921364:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757361.921371:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757361.921377:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757361.921382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757361.921387:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757361.921392:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757362.113355:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757362.113365:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757362.113375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757362.113382:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757362.113385:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757362.113390:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757362.113394:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757362.113396:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757362.113398:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757362.113401:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757362.113480:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757362.113486:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757362.113491:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757362.945282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757362.945290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757362.945293:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757362.945300:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757362.945305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757362.945310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757362.945313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757362.945316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757362.945320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757363.137330:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757363.137341:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757363.137344:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757363.137348:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757363.137355:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757363.137357:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757363.137361:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757363.137364:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757363.137369:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757363.137371:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757363.137397:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757363.137400:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757363.137403:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757363.393339:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757363.393357:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757363.393362:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757363.393368:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757363.393376:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757363.393388:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10418 00000100:00080000:3.0:1787757363.393406:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757363.393412:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757363.393416:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757363.393423:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757363.393428:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757363.393433:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757363.393438:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757363.393442:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757363.393446:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757363.393456:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757363.393463:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757363.393469:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757363.393475:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757363.393482:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393486:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757363.393491:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757363.393508:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757363.393511:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393517:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757363.393522:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757363.393529:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393549:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757363.393568:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659640448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757363.393588:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757363.393597:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393625:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757363.393632:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757363.393639:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757363.393643:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757363.393646:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757363.393652:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757363.393658:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10418 00000100:00080000:3.0:1787757363.393667:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757363.393672:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757363.393675:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757363.393679:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757363.393682:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757363.393686:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757363.393689:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757363.393693:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757363.393694:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757363.393695:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757363.393703:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757363.393712:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:3.0:1787757363.393712:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757363.393719:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757363.393726:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393727:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.393734:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757363.393738:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757363.393741:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757363.393747:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757363.393752:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787757363.393757:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659640448:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757363.393763:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.393767:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757363.393774:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757363.393777:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757363.393805:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.393814:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00100000:3.0:1787757363.393817:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757363.393825:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757363.393828:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393836:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757363.393848:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757363.393852:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393856:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757363.393858:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757363.393860:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757363.393866:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 02000000:00000001:3.0:1787757363.393870:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.393880:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757363.393890:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757363.393908:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659640576/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757363.393926:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757363.393931:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393939:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757363.393945:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757363.393951:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757363.393961:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757363.393964:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757363.393967:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757363.393975:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757363.393976:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757363.393982:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.393988:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757363.393989:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757363.394001:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757363.394005:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10418 00000100:00000001:0.0:1787757363.394009:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305091265 00000100:00080000:3.0:1787757363.394023:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757363.394027:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757363.394035:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757363.394039:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305091265 00000100:00000001:3.0:1787757363.394043:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757363.394049:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:3.0:1787757363.394053:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757363.394056:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787757363.394060:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757363.394063:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757363.394070:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757363.394071:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:3.0:1787757363.394083:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00100000:0.0:1787757363.394084:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757363/real 1787757363] req@ffff8ffb45c31c00 x1874588659640448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757363.394095:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757363.394102:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:0.0:1787757363.394115:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:3.0:1787757363.394122:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787757363.394123:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757363.394131:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757363.394135:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394137:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757363.394142:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757363.394148:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757363.394149:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757363.394156:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757363.394161:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787757363.394165:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659640448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757363.394173:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.394218:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757363.394223:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757363.394227:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.394229:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757363.394238:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757363.394240:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.394243:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757363.394246:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757363.394250:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394262:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757363.394262:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394279:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:3.0:1787757363.394282:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659640704/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757363.394291:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.394301:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757363.394307:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.394316:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757363.394322:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757363.394324:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00080000:0.0:1787757363.394327:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757363.394337:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757363.394342:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757363.394349:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394350:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757363.394365:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659640448:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757363.394365:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757363.394382:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757363.394385:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757363.394388:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757363.394395:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757363.394395:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787757363.394405:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00080000:3.0:1787757363.394406:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757363.394412:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757363.394418:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000020:00000001:0.0:1787757363.394419:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757363.394427:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757363.394433:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00080000:3.0:1787757363.394435:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10418 02000000:00000001:0.0:1787757363.394437:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757363.394443:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757363.394449:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757363.394451:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757363.394455:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757363.394460:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757363.394466:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757363.394472:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757363.394473:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000020:00000001:3.0:1787757363.394476:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757363.394482:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757363.394484:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757363.394493:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:3.0:1787757363.394495:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757363.394501:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757363.394504:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757363.394509:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757363.394510:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757363.394519:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787757363.394521:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757363.394529:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.394530:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757363.394541:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757363.394545:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757363.394548:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757363.394552:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757363.394555:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.394559:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.394562:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757363.394568:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757363.394572:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640576:192.168.201.118@tcp:38:kworker.0 00000008:00000001:3.0:1787757363.394575:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.394583:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394587:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757363.394593:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757363.394595:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757363.394599:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.394602:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.394605:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757363.394612:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757363.394614:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757363.394617:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.394626:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757363.394630:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757363.394643:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659640832/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757363.394649:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757363.394657:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:3.0:1787757363.394658:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757363.394666:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394673:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757363.394675:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757363.394713:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757363.394723:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757363.394728:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757363.394730:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394733:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757363.394737:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394743:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305090574 00000100:00000001:0.0:1787757363.394747:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305091265 00000100:00000001:0.0:1787757363.394750:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305090574 00000100:00000001:0.0:1787757363.394752:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305091265 00000100:00000001:0.0:1787757363.394755:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757363.394758:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394761:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757363.394768:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757363/real 1787757363] req@ffff8ffb45c30a80 x1874588659640576/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757363.394808:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757363.394812:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757363.394818:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757363.394821:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757363.394824:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.394828:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757363.394831:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757363.394840:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659640576/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757363.394855:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.394860:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757363.394865:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.394869:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757363.394872:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.394875:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757363.394898:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757363.394902:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757363.394907:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757363.394911:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757363.394915:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640576:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757363.394921:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757363.394923:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757363.394925:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.394928:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757363.394931:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757363.394934:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757363.394937:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757363.394944:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757363.394947:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757363.394949:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.394952:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757363.394957:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.394965:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757363.394967:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:0.0:1787757363.394972:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757363.394991:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757363.394998:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757363.395001:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757363.395003:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395005:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757363.395009:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395013:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305090571 00000100:00000001:0.0:1787757363.395017:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305091265 00000100:00000001:0.0:1787757363.395019:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305090571 00000100:00000001:0.0:1787757363.395022:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305091265 00000100:00000001:0.0:1787757363.395025:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757363.395027:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395030:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757363.395035:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757363/real 1787757363] req@ffff8ffb45c33b80 x1874588659640704/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757363.395045:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757363.395047:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757363.395049:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757363.395051:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757363.395053:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395055:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757363.395057:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757363.395062:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659640704/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757363.395079:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395082:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757363.395085:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395087:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757363.395088:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395090:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757363.395105:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757363.395108:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757363.395112:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757363.395114:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757363.395118:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757363.395123:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395125:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757363.395127:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757363.395129:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757363.395132:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757363.395134:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757363.395136:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757363.395139:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757363.395141:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757363.395143:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757363.395145:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395147:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757363.395149:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757363.395150:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757363.395153:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757363.395155:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757363.395156:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757363.395158:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757363.395159:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757363.395161:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757363.395163:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395165:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395170:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757363.395173:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395176:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757363.395178:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757363.395180:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757363.395181:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757363.395222:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.395228:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757363.395231:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757363.395235:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757363.395243:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640832:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757363.395250:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757363.395253:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757363.395257:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757363.395260:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757363.395266:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395279:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757363.395283:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:0.0:1787757363.395290:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757363.395313:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757363.395320:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757363.395324:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757363.395326:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395328:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757363.395332:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395336:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305090572 00000100:00000001:0.0:1787757363.395339:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305091265 00000100:00000001:0.0:1787757363.395342:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305090572 00000100:00000001:0.0:1787757363.395345:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305091265 00000100:00000001:0.0:1787757363.395348:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757363.395350:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395352:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757363.395358:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757363/real 1787757363] req@ffff8ffb45c30e00 x1874588659640832/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757363.395376:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757363.395378:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757363.395381:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757363.395383:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757363.395385:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395387:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757363.395389:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757363.395394:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659640832/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757363.395402:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395405:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757363.395408:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395410:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757363.395411:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395413:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757363.395429:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757363.395432:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757363.395436:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757363.395438:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757363.395442:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659640832:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757363.395446:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395449:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757363.395451:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757363.395454:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757363.395458:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757363.395461:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757363.395463:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757363.395467:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757363.395469:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757363.395473:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757363.395476:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395480:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757363.395487:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757363.395490:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757363.395494:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757363.395497:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757363.969379:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757363.969388:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757363.969391:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757363.969397:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757363.969402:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757363.969406:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757363.969409:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757363.969413:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757363.969416:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757364.161343:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757364.161350:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757364.161360:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757364.161363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757364.161460:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757364.161468:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757364.161476:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757364.993278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757364.993286:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757364.993287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757364.993293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757364.993297:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757364.993301:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757364.993304:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757364.993307:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757364.993309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757365.185260:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757365.185266:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757365.185273:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757365.185274:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757365.185357:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757365.185360:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757365.185363:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757366.017278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757366.017285:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757366.017287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757366.017292:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757366.017296:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757366.017300:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757366.017302:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757366.017305:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757366.017307:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757366.209279:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757366.209289:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757366.209297:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757366.209299:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757366.209378:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757366.209383:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757366.209387:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757367.041273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757367.041302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757367.041306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757367.041314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757367.041320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757367.041326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757367.041330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757367.041335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757367.041339:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757367.233296:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757367.233308:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757367.233319:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757367.233321:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757367.233351:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757367.233355:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757367.233359:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757368.065303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757368.065312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757368.065316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757368.065323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757368.065327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757368.065334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757368.065339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757368.065343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757368.065349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757368.257334:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757368.257343:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757368.257351:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757368.257353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757368.257440:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757368.257447:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757368.257452:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757368.513317:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757368.513329:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757368.513333:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757368.513338:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757368.513344:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757368.513353:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10423 00000100:00080000:2.0:1787757368.513366:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757368.513370:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757368.513373:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757368.513378:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757368.513382:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757368.513386:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757368.513389:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757368.513392:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757368.513394:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757368.513399:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757368.513403:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757368.513407:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757368.513411:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757368.513415:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513417:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757368.513420:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757368.513434:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757368.513437:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513441:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757368.513444:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757368.513449:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513463:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757368.513475:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659640960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757368.513489:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757368.513494:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513517:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757368.513521:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757368.513526:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757368.513528:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757368.513530:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757368.513534:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757368.513537:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10423 00000100:00080000:2.0:1787757368.513542:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757368.513545:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757368.513547:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757368.513550:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757368.513551:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757368.513554:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757368.513557:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757368.513560:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757368.513562:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757368.513565:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757368.513568:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757368.513570:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757368.513576:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757368.513579:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513581:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757368.513587:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757368.513591:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513593:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757368.513600:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757368.513602:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513605:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757368.513607:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757368.513610:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513628:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.513635:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:2.0:1787757368.513636:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659641088/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757368.513644:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757368.513651:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757368.513656:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757368.513660:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757368.513661:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513669:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.513672:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757368.513678:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.513679:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787757368.513686:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757368.513717:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757368.513722:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757368.513731:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659640960:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757368.513733:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757368.513740:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757368.513743:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757368.513752:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757368.513753:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757368.513761:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.513766:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:2.0:1787757368.513769:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10423 02000000:00000001:0.0:1787757368.513776:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757368.513782:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757368.513789:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757368.513794:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757368.513800:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:2.0:1787757368.513801:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757368.513807:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000020:00000001:2.0:1787757368.513809:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757368.513815:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787757368.513819:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757368.513826:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757368.513829:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757368.513832:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757368.513838:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757368.513842:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757368.513844:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757368.513851:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757368.513854:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513858:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757368.513862:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757368.513869:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:2.0:1787757368.513876:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513882:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.513889:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787757368.513892:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757368.513897:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787757368.513898:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.513903:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513905:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757368.513912:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787757368.513913:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757368.513920:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.513924:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.513930:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.513932:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305096385 00000100:00100000:2.0:1787757368.513945:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659641216/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757368.513950:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787757368.513965:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305096385 00000100:00000001:2.0:1787757368.513966:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757368.513974:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.513976:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757368.513980:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757368.513986:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757368.513990:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757368.513992:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757368.513994:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757368.513999:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757368.514000:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787757368.514006:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10423 00000100:00000001:0.0:1787757368.514009:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514016:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:2.0:1787757368.514019:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757368.514028:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:0.0:1787757368.514029:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757368/real 1787757368] req@ffff8ffb47a38380 x1874588659640960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757368.514032:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757368.514044:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757368.514050:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757368.514056:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:0.0:1787757368.514061:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:2.0:1787757368.514062:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:0.0:1787757368.514067:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757368.514070:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757368.514076:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757368.514077:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757368.514085:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:2.0:1787757368.514089:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757368.514094:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757368.514096:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757368.514099:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757368.514103:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757368.514105:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514109:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757368.514112:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757368.514122:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787757368.514124:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659640960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757368.514127:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757368.514136:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757368.514144:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757368.514153:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757368.514155:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757368.514158:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514163:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757368.514164:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757368.514171:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757368.514173:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757368.514176:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514179:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757368.514243:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514254:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757368.514256:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659641344/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757368.514261:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757368.514267:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757368.514277:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:2.0:1787757368.514280:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757368.514287:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757368.514288:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757368.514296:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514298:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757368.514309:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659640960:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757368.514318:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514321:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757368.514324:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757368.514328:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757368.514335:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757368.514338:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757368.514340:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757368.514346:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757368.514349:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757368.514354:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757368.514357:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514360:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514369:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757368.514371:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514374:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757368.514376:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757368.514378:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757368.514380:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757368.514382:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.514385:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757368.514388:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757368.514391:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757368.514395:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641088:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757368.514399:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757368.514401:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757368.514403:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.514406:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757368.514410:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514417:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757368.514419:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:0.0:1787757368.514425:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757368.514442:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757368.514448:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757368.514451:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757368.514453:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514455:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757368.514459:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514464:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305095694 00000100:00000001:0.0:1787757368.514467:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305096385 00000100:00000001:0.0:1787757368.514470:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305095694 00000100:00000001:0.0:1787757368.514473:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305096385 00000100:00000001:0.0:1787757368.514476:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757368.514478:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514481:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757368.514486:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757368/real 1787757368] req@ffff8ffb47a38000 x1874588659641088/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757368.514496:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757368.514498:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757368.514501:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757368.514503:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757368.514505:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514507:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757368.514509:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757368.514514:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659641088/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757368.514522:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514525:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757368.514528:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514530:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757368.514533:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514535:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757368.514550:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757368.514553:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757368.514557:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757368.514560:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757368.514564:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641088:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757368.514568:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757368.514570:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757368.514572:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.514574:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757368.514576:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757368.514579:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757368.514582:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757368.514586:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757368.514588:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757368.514590:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.514593:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757368.514596:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514602:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757368.514604:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757368.514608:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757368.514620:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757368.514625:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757368.514627:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757368.514629:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514631:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757368.514634:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514638:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305095691 00000100:00000001:0.0:1787757368.514641:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305096385 00000100:00000001:0.0:1787757368.514644:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305095691 00000100:00000001:0.0:1787757368.514647:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305096385 00000100:00000001:0.0:1787757368.514650:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757368.514652:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514655:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757368.514659:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757368/real 1787757368] req@ffff8ffb47a3b480 x1874588659641216/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757368.514669:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757368.514670:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757368.514672:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757368.514674:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757368.514676:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514678:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757368.514680:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757368.514684:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659641216/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757368.514718:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514721:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757368.514724:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514726:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757368.514728:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514730:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757368.514743:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757368.514746:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757368.514750:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757368.514753:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757368.514758:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757368.514763:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757368.514764:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757368.514766:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.514769:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757368.514771:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757368.514773:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757368.514777:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641344:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757368.514781:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757368.514783:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757368.514785:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757368.514787:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757368.514791:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514797:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757368.514798:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757368.514803:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757368.514816:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757368.514821:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757368.514824:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757368.514825:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514828:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757368.514831:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514835:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305095691 00000100:00000001:0.0:1787757368.514838:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305096385 00000100:00000001:0.0:1787757368.514841:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305095691 00000100:00000001:0.0:1787757368.514844:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305096385 00000100:00000001:0.0:1787757368.514847:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757368.514849:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.514852:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757368.514856:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757368/real 1787757368] req@ffff8ffb47a39180 x1874588659641344/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757368.514866:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757368.514868:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757368.514870:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757368.514872:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757368.514874:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514876:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757368.514878:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757368.514882:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659641344/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757368.514900:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514902:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757368.514905:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514907:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757368.514909:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514911:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757368.514923:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757368.514926:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757368.514929:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757368.514932:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757368.514935:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641344:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757368.514940:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514942:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757368.514944:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757368.514946:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757368.514949:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757368.514950:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757368.514952:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757368.514955:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757368.514957:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757368.514959:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757368.514961:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514963:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757368.514965:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757368.514966:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757368.514969:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757368.514970:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757368.514972:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757368.514974:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757368.514975:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757368.514977:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757368.514979:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.514981:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757368.514982:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757368.514984:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757368.514987:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757368.514988:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757368.514990:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757368.514992:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757368.514993:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757368.514997:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757368.514998:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.515000:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757368.515004:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757368.515005:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757368.515008:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757368.515010:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757369.089296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757369.089304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757369.089307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757369.089313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757369.089318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757369.089323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757369.089327:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757369.089330:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757369.089333:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757369.281323:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757369.281333:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757369.281344:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757369.281347:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757369.281400:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757369.281409:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757369.281416:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757370.113313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757370.113322:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757370.113325:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757370.113332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757370.113337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757370.113342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757370.113345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757370.113349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757370.113352:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757370.305342:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757370.305350:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757370.305357:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757370.305359:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757370.305509:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757370.305515:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757370.305521:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757371.137264:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757371.137275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757371.137278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757371.137285:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757371.137292:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757371.137298:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757371.137303:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757371.137308:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757371.137314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757371.329323:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757371.329332:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757371.329341:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757371.329346:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757371.329350:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757371.329355:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757371.329358:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757371.329361:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757371.329363:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757371.329366:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757371.329394:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757371.329398:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757371.329401:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757372.161304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757372.161319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757372.161325:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757372.161336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757372.161344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757372.161351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757372.161357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757372.161363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757372.161369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757372.353342:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757372.353353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757372.353362:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757372.353363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757372.353439:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757372.353448:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757372.353454:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757373.185278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757373.185291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757373.185294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757373.185304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757373.185312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757373.185318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757373.185323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757373.185328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757373.185333:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757373.377340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757373.377351:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757373.377355:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757373.377359:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757373.377365:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757373.377368:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757373.377371:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757373.377374:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757373.377379:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757373.377381:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757373.377511:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757373.377518:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757373.377522:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757373.633347:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757373.633362:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757373.633367:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757373.633372:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757373.633378:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757373.633390:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10428 00000100:00080000:2.0:1787757373.633407:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757373.633412:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757373.633418:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757373.633424:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757373.633428:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757373.633433:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757373.633438:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757373.633442:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757373.633445:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757373.633452:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757373.633456:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.633460:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757373.633464:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757373.633468:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633471:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757373.633474:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.633487:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757373.633489:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633493:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757373.633496:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757373.633501:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633515:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757373.633526:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659641472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757373.633539:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757373.633545:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633565:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757373.633570:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757373.633574:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757373.633576:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757373.633578:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757373.633582:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757373.633585:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10428 00000100:00080000:2.0:1787757373.633590:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757373.633595:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757373.633620:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757373.633624:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757373.633626:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757373.633630:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757373.633635:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757373.633639:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757373.633643:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757373.633651:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757373.633657:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.633660:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757373.633668:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757373.633671:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757373.633679:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757373.633680:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.633688:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757373.633689:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757373.633701:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757373.633706:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.633711:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757373.633712:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.633719:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.633723:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757373.633728:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633731:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757373.633734:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757373.633734:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757373.633759:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757373.633761:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.633770:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633777:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757373.633783:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659641472:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1787757373.633794:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659641600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757373.633795:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757373.633804:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757373.633810:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.633819:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757373.633821:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757373.633826:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.633829:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633832:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757373.633839:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757373.633844:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757373.633847:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757373.633849:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757373.633854:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757373.633855:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757373.633861:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00080000:2.0:1787757373.633865:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10428 00000100:00080000:2.0:1787757373.633873:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757373.633875:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757373.633877:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757373.633884:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787757373.633885:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757373.633893:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757373.633897:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757373.633900:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757373.633903:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757373.633905:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757373.633911:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757373.633915:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.633917:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757373.633921:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757373.633924:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633926:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757373.633931:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757373.633934:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633936:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.633942:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757373.633945:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633967:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000800:00000100:0.0:1787757373.633970:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:2.0:1787757373.633971:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757373.633976:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.633985:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757373.633995:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659641728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757373.633996:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757373.634008:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757373.634009:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757373.634015:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634016:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634023:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757373.634024:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757373.634031:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757373.634034:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757373.634037:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757373.634043:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757373.634045:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305101505 00000100:00080000:2.0:1787757373.634046:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757373.634055:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757373.634059:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10428 00000100:00080000:2.0:1787757373.634064:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757373.634065:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757373.634067:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757373.634073:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757373.634077:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757373.634078:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305101505 00000020:00000001:2.0:1787757373.634080:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757373.634086:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757373.634090:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:2.0:1787757373.634091:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757373.634100:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757373.634102:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757373.634106:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757373.634109:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.634111:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757373.634113:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757373.634115:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757373.634122:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.634124:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634133:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787757373.634134:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757373.634142:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757373.634145:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757373.634150:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757373/real 1787757373] req@ffff8ffb47a3a300 x1874588659641472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757373.634150:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757373.634162:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757373.634166:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.634174:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757373.634178:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000400:00000001:0.0:1787757373.634179:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:2.0:1787757373.634180:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757373.634186:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757373.634193:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659641856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787757373.634218:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757373.634227:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757373.634232:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757373.634234:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634238:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757373.634240:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757373.634249:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659641472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757373.634255:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757373.634264:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634267:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757373.634270:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634281:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757373.634289:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634293:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757373.634296:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634301:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757373.634305:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757373.634311:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757373.634319:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757373.634324:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757373.634333:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659641472:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757373.634344:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634348:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757373.634352:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757373.634356:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757373.634363:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757373.634366:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757373.634368:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757373.634374:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757373.634376:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757373.634380:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757373.634382:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634385:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634393:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757373.634396:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634399:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757373.634401:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757373.634403:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757373.634405:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757373.634407:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.634410:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757373.634413:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757373.634415:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757373.634419:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641600:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757373.634423:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757373.634426:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757373.634428:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.634430:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757373.634434:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634443:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757373.634445:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:0.0:1787757373.634450:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757373.634467:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757373.634474:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757373.634477:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757373.634479:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634481:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757373.634484:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634490:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305100814 00000100:00000001:0.0:1787757373.634493:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305101505 00000100:00000001:0.0:1787757373.634496:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305100814 00000100:00000001:0.0:1787757373.634499:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305101505 00000100:00000001:0.0:1787757373.634502:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757373.634505:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634507:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757373.634512:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757373/real 1787757373] req@ffff8ffb47a38700 x1874588659641600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757373.634523:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757373.634525:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757373.634528:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757373.634530:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757373.634532:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634534:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757373.634535:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757373.634540:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659641600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757373.634548:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634551:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757373.634554:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634556:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757373.634557:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634559:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757373.634574:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757373.634577:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757373.634581:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757373.634583:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757373.634587:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641600:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757373.634591:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757373.634593:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757373.634594:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.634621:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757373.634624:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757373.634627:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757373.634631:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757373.634637:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757373.634639:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757373.634642:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.634645:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757373.634651:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634661:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757373.634663:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757373.634669:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757373.634692:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757373.634701:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757373.634705:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757373.634708:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634711:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757373.634716:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634722:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305100811 00000100:00000001:0.0:1787757373.634727:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305101505 00000100:00000001:0.0:1787757373.634731:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305100811 00000100:00000001:0.0:1787757373.634735:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305101505 00000100:00000001:0.0:1787757373.634739:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757373.634742:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634744:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757373.634750:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757373/real 1787757373] req@ffff8ffb47a39180 x1874588659641728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757373.634777:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757373.634779:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757373.634784:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757373.634786:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757373.634788:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634790:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757373.634792:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757373.634797:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659641728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757373.634814:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634817:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757373.634820:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634822:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757373.634824:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.634826:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757373.634838:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757373.634841:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757373.634844:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757373.634847:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757373.634850:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757373.634855:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757373.634857:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757373.634859:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.634862:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757373.634865:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757373.634867:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757373.634871:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641856:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757373.634875:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757373.634877:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757373.634879:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757373.634881:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757373.634885:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634891:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757373.634893:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757373.634899:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757373.634937:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757373.634945:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757373.634949:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757373.634952:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634955:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757373.634961:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634966:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305100811 00000100:00000001:0.0:1787757373.634971:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305101505 00000100:00000001:0.0:1787757373.634976:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305100811 00000100:00000001:0.0:1787757373.634980:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305101505 00000100:00000001:0.0:1787757373.634984:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757373.634989:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.634992:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757373.635000:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757373/real 1787757373] req@ffff8ffb47a3b480 x1874588659641856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757373.635014:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757373.635016:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757373.635019:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757373.635021:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757373.635023:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635025:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757373.635027:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757373.635032:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659641856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757373.635040:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635043:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757373.635046:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635048:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757373.635049:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635051:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757373.635063:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757373.635066:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757373.635069:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757373.635072:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757373.635076:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659641856:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757373.635081:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635083:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757373.635085:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757373.635087:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757373.635090:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757373.635091:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757373.635093:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757373.635096:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757373.635098:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757373.635102:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757373.635103:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635105:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757373.635107:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757373.635109:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757373.635111:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757373.635113:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757373.635114:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757373.635116:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757373.635117:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757373.635120:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757373.635121:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635123:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757373.635124:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757373.635126:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757373.635128:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757373.635129:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757373.635130:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757373.635132:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757373.635133:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757373.635135:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757373.635137:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635139:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757373.635143:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757373.635144:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757373.635147:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757373.635149:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757374.209306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757374.209315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757374.209318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757374.209325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757374.209329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757374.209334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757374.209338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757374.209342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757374.209345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757374.401465:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757374.401476:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757374.401486:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757374.401489:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757374.401607:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757374.401616:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757374.401623:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757375.233307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757375.233316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757375.233320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757375.233326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757375.233331:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757375.233336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757375.233339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757375.233343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757375.233346:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757375.425390:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757375.425401:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757375.425409:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757375.425411:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757375.425446:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757375.425450:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757375.425454:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757376.257317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757376.257328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757376.257331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757376.257339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757376.257344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757376.257348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757376.257352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757376.257355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757376.257359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757376.449410:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757376.449421:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757376.449431:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757376.449433:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757376.449506:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757376.449514:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757376.449520:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757377.281334:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757377.281343:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757377.281347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757377.281352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757377.281357:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757377.281362:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757377.281366:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757377.281370:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757377.281373:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757377.473359:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757377.473367:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757377.473374:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757377.473376:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757377.473460:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757377.473467:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757377.473474:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757378.305312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757378.305321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757378.305324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757378.305331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757378.305336:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757378.305341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.305345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757378.305348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.305351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757378.497376:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757378.497385:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757378.497394:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757378.497395:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757378.497579:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757378.497586:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757378.497592:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757378.753378:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757378.753392:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757378.753396:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757378.753401:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757378.753408:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757378.753420:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10433 00000100:00080000:2.0:1787757378.753435:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757378.753440:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757378.753443:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757378.753450:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757378.753455:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757378.753460:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757378.753464:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757378.753467:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757378.753470:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757378.753476:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757378.753479:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757378.753482:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757378.753488:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757378.753493:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.753497:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757378.753502:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757378.753541:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757378.753546:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.753551:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757378.753559:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757378.753567:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.753588:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757378.753608:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659641984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757378.753629:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757378.753637:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.753666:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757378.753671:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757378.753677:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757378.753680:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757378.753683:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757378.753689:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757378.753696:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10433 00000100:00080000:2.0:1787757378.753707:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757378.753712:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757378.753717:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757378.753722:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757378.753725:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757378.753729:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757378.753733:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757378.753742:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757378.753744:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757378.753746:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757378.753760:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757378.753763:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757378.753768:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757378.753775:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757378.753777:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787757378.753790:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757378.753791:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787757378.753803:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.753806:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.753808:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757378.753815:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757378.753822:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.753824:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757378.753827:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.753832:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757378.753836:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757378.753840:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757378.753841:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.753860:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757378.753864:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757378.753869:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757378.753873:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659641984:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757378.753880:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.753887:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:2.0:1787757378.753895:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659642112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757378.753898:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757378.753904:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.753921:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757378.753935:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757378.753940:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.753941:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.753960:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757378.753969:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757378.753975:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757378.753978:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757378.753980:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757378.753984:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757378.753986:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757378.753990:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00080000:2.0:1787757378.753999:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10433 00000100:00000001:1.0:1787757378.754004:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787757378.754007:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757378.754014:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757378.754018:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757378.754021:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757378.754025:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757378.754029:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757378.754033:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757378.754036:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757378.754038:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757378.754044:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757378.754047:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757378.754050:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757378.754057:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757378.754061:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.754063:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757378.754066:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757378.754069:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.754072:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757378.754085:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757378.754087:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.754091:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757378.754093:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757378.754095:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757378.754098:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757378.754101:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757378.754117:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659642240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757378.754130:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757378.754135:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757378.754143:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754145:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757378.754151:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754156:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757378.754161:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757378.754167:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:2.0:1787757378.754174:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757378.754241:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757378.754243:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757378.754248:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757378.754255:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305106626 00000100:00000001:2.0:1787757378.754257:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757378.754278:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10433 00000100:00000001:1.0:1787757378.754283:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:2.0:1787757378.754290:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757378.754296:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305106626 00000100:00000001:2.0:1787757378.754298:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757378.754309:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757378.754312:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:2.0:1787757378.754330:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757378.754334:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787757378.754338:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757378.754343:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757378.754345:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757378.754355:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:2.0:1787757378.754357:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757378.754365:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757378.754372:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757378.754374:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757378/real 1787757378] req@ffff8ffb47a38000 x1874588659641984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757378.754394:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757378.754420:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757378.754427:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000400:00000001:1.0:1787757378.754429:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757378.754439:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:2.0:1787757378.754442:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757378.754452:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754454:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757378.754465:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757378.754468:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000008:00100000:2.0:1787757378.754470:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757378.754476:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.754478:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757378.754484:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757378.754486:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757378.754491:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757378.754493:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757378.754495:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757378.754500:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754505:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757378.754531:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757378.754538:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757378.754550:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659641984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757378.754563:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659642368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757378.754609:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757378.754629:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757378.754630:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757378.754638:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754642:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754648:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757378.754648:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754659:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754665:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757378.754668:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757378.754674:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757378.754682:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757378.754687:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757378.754695:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659641984:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757378.754708:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754712:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757378.754716:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757378.754720:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757378.754728:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757378.754733:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757378.754736:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757378.754742:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757378.754744:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757378.754749:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757378.754751:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754755:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754766:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757378.754769:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754772:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757378.754774:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757378.754777:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757378.754779:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757378.754781:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.754785:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757378.754788:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757378.754791:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757378.754796:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642112:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757378.754801:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757378.754804:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757378.754806:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.754809:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757378.754813:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754823:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757378.754825:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:1.0:1787757378.754832:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757378.754854:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757378.754860:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757378.754863:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757378.754866:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754868:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757378.754872:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754878:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305105934 00000100:00000001:1.0:1787757378.754883:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305106626 00000100:00000001:1.0:1787757378.754886:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305105934 00000100:00000001:1.0:1787757378.754889:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305106626 00000100:00000001:1.0:1787757378.754892:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757378.754895:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.754897:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757378.754903:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757378/real 1787757378] req@ffff8ffb47a38380 x1874588659642112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757378.754914:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757378.754915:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757378.754919:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757378.754921:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757378.754923:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754925:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757378.754927:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757378.754931:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659642112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757378.754941:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754943:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757378.754946:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754948:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757378.754950:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.754952:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757378.754975:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757378.754978:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757378.754982:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757378.754984:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757378.754988:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642112:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757378.754996:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757378.754998:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757378.755000:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.755004:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757378.755006:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757378.755009:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757378.755012:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757378.755017:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757378.755019:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757378.755021:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.755023:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757378.755027:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755033:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757378.755035:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757378.755040:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757378.755052:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757378.755057:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757378.755060:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757378.755062:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755064:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757378.755067:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755071:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305105931 00000100:00000001:1.0:1787757378.755074:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305106626 00000100:00000001:1.0:1787757378.755077:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305105931 00000100:00000001:1.0:1787757378.755080:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305106626 00000100:00000001:1.0:1787757378.755083:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757378.755086:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755090:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757378.755098:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757378/real 1787757378] req@ffff8ffb47a38a80 x1874588659642240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757378.755115:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757378.755118:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757378.755124:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757378.755127:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757378.755129:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755133:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757378.755135:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757378.755143:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659642240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757378.755158:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755162:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757378.755167:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755171:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757378.755173:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755177:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757378.755242:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757378.755246:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757378.755252:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757378.755257:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757378.755265:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757378.755272:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757378.755275:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757378.755278:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.755281:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757378.755285:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757378.755289:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757378.755295:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757378.755300:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757378.755304:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757378.755307:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757378.755309:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757378.755314:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755326:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757378.755329:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757378.755335:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757378.755356:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757378.755362:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757378.755365:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757378.755368:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755370:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757378.755373:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755378:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305105932 00000100:00000001:1.0:1787757378.755381:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305106626 00000100:00000001:1.0:1787757378.755385:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305105932 00000100:00000001:1.0:1787757378.755387:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305106626 00000100:00000001:1.0:1787757378.755390:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757378.755393:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755395:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757378.755400:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757378/real 1787757378] req@ffff8ffb47a3a680 x1874588659642368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757378.755409:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757378.755410:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757378.755413:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757378.755415:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757378.755417:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755419:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757378.755421:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757378.755426:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659642368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757378.755434:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755437:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757378.755440:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755442:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757378.755443:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755445:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757378.755465:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757378.755468:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757378.755472:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757378.755475:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757378.755480:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757378.755487:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755491:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757378.755494:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757378.755497:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757378.755501:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757378.755504:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757378.755526:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757378.755532:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757378.755534:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757378.755539:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757378.755541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755545:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757378.755548:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757378.755551:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757378.755556:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757378.755558:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757378.755560:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757378.755566:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757378.755568:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757378.755572:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757378.755574:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755578:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757378.755580:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757378.755583:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757378.755586:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757378.755589:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757378.755591:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757378.755594:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757378.755597:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757378.755602:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757378.755604:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755608:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757378.755614:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757378.755616:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757378.755619:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757378.755621:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757379.329329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757379.329343:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757379.329346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757379.329355:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757379.329360:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757379.329367:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757379.329370:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757379.329375:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757379.329379:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757379.521375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757379.521387:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757379.521397:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757379.521399:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757379.521426:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757379.521429:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757379.521442:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757380.353275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757380.353284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757380.353286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757380.353292:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757380.353296:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757380.353300:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757380.353302:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757380.353305:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757380.353308:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757380.545350:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757380.545358:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757380.545369:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757380.545371:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757380.545505:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757380.545518:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757380.545528:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757381.377317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757381.377329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757381.377333:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757381.377340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757381.377346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757381.377352:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757381.377356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757381.377360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757381.377363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757381.569324:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757381.569333:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757381.569342:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757381.569347:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757381.569351:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757381.569356:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757381.569361:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757381.569365:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757381.569368:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757381.569371:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757381.569522:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757381.569529:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757381.569534:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757382.401353:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757382.401366:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757382.401370:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757382.401379:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757382.401384:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757382.401391:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757382.401394:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757382.401398:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757382.401402:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757382.593303:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757382.593314:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757382.593324:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757382.593327:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757382.593534:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757382.593545:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757382.593552:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757383.425386:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757383.425401:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757383.425404:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757383.425412:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757383.425417:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757383.425423:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.425485:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757383.425489:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.425493:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757383.617329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757383.617338:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757383.617341:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757383.617345:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757383.617351:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757383.617354:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757383.617357:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757383.617373:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757383.617378:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757383.617380:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757383.617408:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757383.617413:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757383.617418:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757383.873361:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757383.873373:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757383.873376:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757383.873381:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757383.873388:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757383.873400:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10438 00000100:00080000:2.0:1787757383.873440:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757383.873448:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757383.873452:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757383.873459:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757383.873464:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757383.873472:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757383.873478:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757383.873483:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757383.873486:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757383.873495:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757383.873501:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757383.873506:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757383.873513:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757383.873519:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.873523:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757383.873528:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757383.873546:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757383.873548:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.873553:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757383.873556:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757383.873562:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.873577:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757383.873590:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659642496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757383.873604:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757383.873610:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.873644:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757383.873649:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757383.873654:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757383.873656:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757383.873659:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757383.873665:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757383.873670:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10438 00000100:00080000:2.0:1787757383.873679:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757383.873685:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757383.873688:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757383.873692:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757383.873696:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757383.873701:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757383.873706:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:2.0:1787757383.873716:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757383.873717:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757383.873725:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757383.873728:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757383.873733:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757383.873736:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757383.873742:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757383.873752:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757383.873757:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757383.873762:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.873767:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757383.873772:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757383.873777:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757383.873783:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757383.873793:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757383.873795:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659642496:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757383.873798:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757383.873809:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757383.873811:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:2.0:1787757383.873815:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.873821:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.873822:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757383.873828:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757383.873830:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757383.873834:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.873841:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757383.873841:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757383.873848:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757383.873854:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.873858:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.873878:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.873890:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:2.0:1787757383.873891:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659642624/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757383.873900:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:2.0:1787757383.873915:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757383.873917:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757383.873920:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.873929:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757383.873933:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757383.873938:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757383.873940:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757383.873943:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757383.873948:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757383.873951:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10438 00000100:00080000:2.0:1787757383.873959:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757383.873963:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757383.873967:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757383.873970:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757383.873972:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757383.873974:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757383.873977:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757383.873979:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757383.873982:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:1.0:1787757383.873991:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787757383.873997:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757383.874003:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757383.874007:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757383.874012:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757383.874013:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787757383.874016:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.874021:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757383.874024:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00100000:2.0:1787757383.874026:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757383.874033:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.874035:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757383.874041:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757383.874043:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.874045:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757383.874046:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757383.874048:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757383.874050:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874058:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757383.874065:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874071:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757383.874077:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659642752/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757383.874081:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305111745 00000100:00000001:2.0:1787757383.874098:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757383.874102:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874103:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757383.874106:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757383.874113:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757383.874115:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305111745 00000100:00080000:2.0:1787757383.874119:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757383.874126:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757383.874128:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:2.0:1787757383.874129:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757383.874139:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757383.874140:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757383.874149:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757383.874150:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10438 00000100:00000001:1.0:1787757383.874160:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:2.0:1787757383.874162:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757383.874168:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757383.874172:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757383.874175:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757383/real 1787757383] req@ffff8ffb47a39880 x1874588659642496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787757383.874177:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757383.874183:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757383.874233:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787757383.874238:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:2.0:1787757383.874240:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787757383.874245:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757383.874247:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757383.874252:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757383.874255:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:2.0:1787757383.874262:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757383.874264:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757383.874271:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757383.874272:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874278:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:2.0:1787757383.874280:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757383.874285:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757383.874288:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757383.874299:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757383.874300:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659642496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757383.874303:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757383.874310:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757383.874320:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.874329:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874332:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757383.874337:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757383.874340:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:2.0:1787757383.874343:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874350:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757383.874351:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757383.874355:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757383.874358:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757383.874363:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874369:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:2.0:1787757383.874376:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659642880/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757383.874377:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874385:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757383.874391:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757383.874401:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757383.874413:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757383.874438:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757383.874442:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757383.874449:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757383.874454:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659642496:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757383.874460:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874465:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874472:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757383.874476:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757383.874480:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757383.874487:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757383.874490:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757383.874492:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757383.874498:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757383.874501:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757383.874507:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757383.874509:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874512:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874521:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757383.874524:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874529:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757383.874533:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757383.874535:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757383.874537:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757383.874540:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.874543:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757383.874546:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757383.874549:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757383.874554:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642624:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757383.874559:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757383.874562:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757383.874564:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.874566:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757383.874570:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874579:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757383.874581:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:1.0:1787757383.874586:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757383.874605:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757383.874612:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757383.874615:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757383.874617:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874619:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757383.874623:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874628:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305111054 00000100:00000001:1.0:1787757383.874632:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305111745 00000100:00000001:1.0:1787757383.874635:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305111054 00000100:00000001:1.0:1787757383.874638:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305111745 00000100:00000001:1.0:1787757383.874641:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757383.874644:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874647:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757383.874652:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757383/real 1787757383] req@ffff8ffb47a39500 x1874588659642624/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757383.874663:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757383.874665:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757383.874668:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757383.874670:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757383.874672:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874674:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757383.874676:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757383.874681:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659642624/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757383.874690:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874693:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757383.874696:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874698:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757383.874699:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874701:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757383.874717:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757383.874720:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757383.874724:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757383.874727:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757383.874731:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642624:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757383.874735:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757383.874737:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757383.874739:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.874741:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757383.874744:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757383.874747:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757383.874750:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757383.874755:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757383.874757:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757383.874758:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.874760:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757383.874764:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874769:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757383.874771:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757383.874775:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757383.874787:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757383.874791:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757383.874794:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757383.874796:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874798:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757383.874801:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874805:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305111051 00000100:00000001:1.0:1787757383.874808:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305111745 00000100:00000001:1.0:1787757383.874811:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305111051 00000100:00000001:1.0:1787757383.874814:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305111745 00000100:00000001:1.0:1787757383.874817:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757383.874819:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874821:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757383.874826:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757383/real 1787757383] req@ffff8ffb47a3a680 x1874588659642752/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757383.874835:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757383.874836:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757383.874838:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757383.874840:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757383.874842:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874844:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757383.874845:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757383.874851:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659642752/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757383.874860:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874863:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757383.874865:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874867:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757383.874869:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.874871:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757383.874885:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757383.874888:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757383.874892:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757383.874894:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757383.874898:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757383.874902:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757383.874904:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757383.874906:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.874909:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757383.874911:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757383.874913:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757383.874917:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642880:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757383.874920:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757383.874922:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757383.874923:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757383.874925:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757383.874929:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874933:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757383.874935:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757383.874939:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757383.874950:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757383.874955:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757383.874958:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757383.874960:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874963:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757383.874965:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874969:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305111051 00000100:00000001:1.0:1787757383.874972:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305111745 00000100:00000001:1.0:1787757383.874976:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305111051 00000100:00000001:1.0:1787757383.874979:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305111745 00000100:00000001:1.0:1787757383.874981:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757383.874984:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.874986:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757383.874990:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757383/real 1787757383] req@ffff8ffb47a38a80 x1874588659642880/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757383.874999:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757383.875000:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757383.875003:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757383.875005:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757383.875007:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875009:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757383.875011:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757383.875015:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659642880/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757383.875023:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875025:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757383.875028:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875030:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757383.875031:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875033:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757383.875047:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757383.875050:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757383.875053:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757383.875055:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757383.875060:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659642880:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757383.875065:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875068:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757383.875069:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757383.875071:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757383.875074:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757383.875075:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757383.875077:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757383.875080:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757383.875082:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757383.875084:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757383.875086:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875088:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757383.875090:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757383.875091:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757383.875094:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757383.875096:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757383.875097:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757383.875099:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757383.875100:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757383.875103:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757383.875105:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875107:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757383.875108:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757383.875109:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757383.875111:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757383.875113:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757383.875114:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757383.875116:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757383.875118:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757383.875121:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757383.875123:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875125:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757383.875129:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757383.875131:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757383.875133:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757383.875135:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757384.449307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757384.449316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757384.449319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757384.449327:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757384.449332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757384.449336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757384.449339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757384.449342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757384.449346:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757384.641356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757384.641367:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757384.641379:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757384.641383:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757384.641487:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757384.641493:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757384.641498:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757385.473297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757385.473306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757385.473309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757385.473316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757385.473320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757385.473325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757385.473329:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757385.473332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757385.473336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757385.665432:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757385.665442:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757385.665452:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757385.665454:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757385.665566:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757385.665573:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757385.665577:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757386.497290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757386.497301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757386.497306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757386.497314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757386.497321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757386.497326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757386.497331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757386.497336:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757386.497341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757386.689316:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757386.689326:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757386.689336:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757386.689339:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757386.689429:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757386.689435:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757386.689442:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757387.521282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757387.521291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757387.521294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757387.521301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757387.521305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757387.521310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757387.521314:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757387.521317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757387.521321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757387.713299:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757387.713307:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757387.713316:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757387.713319:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757387.713393:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757387.713398:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757387.713403:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757388.545274:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757388.545284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757388.545287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757388.545293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757388.545298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757388.545304:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.545307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757388.545310:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.545313:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757388.737396:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757388.737405:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757388.737411:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757388.737413:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757388.737495:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757388.737500:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757388.737505:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757388.993303:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757388.993317:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757388.993320:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757388.993325:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757388.993353:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757388.993365:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10443 00000100:00080000:2.0:1787757388.993382:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757388.993388:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757388.993392:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757388.993399:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757388.993404:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757388.993410:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757388.993416:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757388.993420:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757388.993423:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757388.993432:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757388.993439:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757388.993444:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757388.993450:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757388.993457:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993461:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757388.993466:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757388.993480:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757388.993483:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993487:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757388.993491:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757388.993497:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993511:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757388.993524:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659643008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757388.993539:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757388.993545:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993568:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757388.993572:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757388.993577:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757388.993579:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757388.993582:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757388.993586:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757388.993589:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10443 00000100:00080000:2.0:1787757388.993596:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757388.993599:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757388.993601:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757388.993604:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757388.993606:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757388.993609:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757388.993611:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757388.993613:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757388.993615:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757388.993631:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757388.993635:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757388.993638:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757388.993645:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757388.993648:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993651:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757388.993658:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757388.993663:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.993665:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757388.993667:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.993678:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757388.993679:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757388.993688:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.993690:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757388.993691:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757388.993696:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757388.993702:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787757388.993702:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.993713:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993715:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757388.993728:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:2.0:1787757388.993730:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659643136/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757388.993740:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757388.993748:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757388.993751:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993756:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993765:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757388.993766:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659643008:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787757388.993772:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757388.993782:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787757388.993783:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757388.993790:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:1.0:1787757388.993793:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:2.0:1787757388.993794:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757388.993802:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993804:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757388.993810:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:2.0:1787757388.993816:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10443 00000100:00080000:2.0:1787757388.993824:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757388.993825:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993827:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757388.993834:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757388.993839:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757388.993843:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757388.993847:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757388.993852:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:2.0:1787757388.993853:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757388.993858:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:2.0:1787757388.993865:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757388.993868:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757388.993872:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757388.993885:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757388.993891:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757388.993895:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757388.993902:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757388.993908:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993912:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757388.993919:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757388.993923:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:2.0:1787757388.993930:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993938:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.993943:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757388.993951:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757388.993954:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.993962:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787757388.993967:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757388.993974:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.993975:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.993978:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757388.993986:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757388.993990:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305116865 02000000:00000001:2.0:1787757388.993991:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.994005:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994015:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:2.0:1787757388.994018:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659643264/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757388.994033:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305116865 00000100:00000001:2.0:1787757388.994035:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757388.994042:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.994049:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994054:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:2.0:1787757388.994055:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757388.994067:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757388.994071:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757388.994074:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757388.994075:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757388.994079:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757388.994088:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10443 00000100:00000001:1.0:1787757388.994090:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757388.994097:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757388.994100:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787757388.994104:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757388.994109:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757388.994113:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757388.994116:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757388.994120:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:1.0:1787757388.994123:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757388/real 1787757388] req@ffff8ffb47a38380 x1874588659643008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787757388.994124:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757388.994131:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757388.994134:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757388.994149:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757388.994159:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757388.994166:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757388.994172:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757388.994181:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.994230:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000400:00000001:1.0:1787757388.994237:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000008:00100000:2.0:1787757388.994237:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:1.0:1787757388.994244:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000008:00000001:2.0:1787757388.994246:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.994252:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994254:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:2.0:1787757388.994259:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757388.994262:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:2.0:1787757388.994264:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994268:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757388.994271:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757388.994277:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:2.0:1787757388.994278:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757388.994285:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757388.994287:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757388.994300:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757388.994302:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659643008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757388.994317:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659643392/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757388.994353:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994364:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757388.994368:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757388.994371:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757388.994378:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994379:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757388.994384:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994390:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757388.994393:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994395:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757388.994406:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757388.994414:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757388.994418:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757388.994424:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659643008:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757388.994431:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994435:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757388.994439:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757388.994443:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757388.994449:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757388.994452:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757388.994455:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757388.994461:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757388.994463:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757388.994468:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757388.994470:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994473:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994482:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757388.994485:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994488:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757388.994490:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757388.994492:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757388.994494:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757388.994497:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757388.994500:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757388.994502:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757388.994505:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757388.994510:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643136:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757388.994514:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757388.994517:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757388.994519:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757388.994521:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757388.994526:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994535:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757388.994537:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:1.0:1787757388.994543:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757388.994563:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757388.994570:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757388.994573:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757388.994575:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994578:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757388.994582:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994588:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305116174 00000100:00000001:1.0:1787757388.994592:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305116865 00000100:00000001:1.0:1787757388.994595:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305116174 00000100:00000001:1.0:1787757388.994598:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305116865 00000100:00000001:1.0:1787757388.994601:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757388.994603:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994606:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757388.994611:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757388/real 1787757388] req@ffff8ffb47a38000 x1874588659643136/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757388.994638:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757388.994641:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757388.994646:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757388.994649:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757388.994652:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994655:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757388.994658:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757388.994666:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659643136/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757388.994680:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994685:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757388.994690:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994694:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757388.994696:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994700:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757388.994724:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757388.994731:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757388.994737:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757388.994742:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757388.994747:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643136:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757388.994753:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757388.994755:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757388.994757:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757388.994760:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757388.994763:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757388.994765:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757388.994769:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757388.994773:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757388.994776:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757388.994779:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757388.994781:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757388.994786:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994794:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757388.994796:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787757388.994802:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757388.994819:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757388.994826:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757388.994829:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757388.994831:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994833:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757388.994837:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994841:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305116171 00000100:00000001:1.0:1787757388.994844:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305116865 00000100:00000001:1.0:1787757388.994847:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305116171 00000100:00000001:1.0:1787757388.994850:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305116865 00000100:00000001:1.0:1787757388.994853:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757388.994855:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994858:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757388.994863:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757388/real 1787757388] req@ffff8ffb47a3b480 x1874588659643264/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757388.994872:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757388.994873:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757388.994876:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757388.994878:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757388.994880:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994882:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757388.994884:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757388.994889:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659643264/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757388.994897:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994900:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757388.994904:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994906:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757388.994908:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.994910:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757388.994923:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757388.994926:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757388.994930:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757388.994933:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757388.994937:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757388.994941:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757388.994943:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757388.994945:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757388.994947:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757388.994949:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757388.994951:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757388.994954:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643392:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757388.994958:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757388.994961:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757388.994963:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757388.994965:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757388.994969:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.994974:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757388.994976:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787757388.994981:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757388.994992:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757388.994997:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757388.994999:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757388.995001:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.995003:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757388.995007:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.995011:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305116171 00000100:00000001:1.0:1787757388.995014:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305116865 00000100:00000001:1.0:1787757388.995017:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305116171 00000100:00000001:1.0:1787757388.995020:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305116865 00000100:00000001:1.0:1787757388.995023:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757388.995026:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.995028:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757388.995033:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757388/real 1787757388] req@ffff8ffb47a39180 x1874588659643392/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757388.995041:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757388.995043:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757388.995046:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757388.995048:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757388.995050:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995052:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757388.995054:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757388.995058:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659643392/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757388.995066:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995069:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757388.995071:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995073:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757388.995075:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995077:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757388.995089:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757388.995092:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757388.995095:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757388.995098:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757388.995101:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643392:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757388.995105:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995108:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757388.995110:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757388.995112:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757388.995116:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757388.995118:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757388.995120:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757388.995123:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757388.995125:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757388.995128:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757388.995129:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995131:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757388.995133:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757388.995135:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757388.995138:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757388.995140:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757388.995141:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757388.995143:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757388.995145:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757388.995147:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757388.995149:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995151:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757388.995152:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757388.995153:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757388.995155:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757388.995156:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757388.995158:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757388.995159:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757388.995161:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757388.995164:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757388.995165:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995167:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757388.995172:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757388.995174:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757388.995176:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757388.995178:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757389.569337:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757389.569354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757389.569359:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757389.569371:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757389.569380:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757389.569388:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757389.569393:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757389.569399:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757389.569407:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757389.761401:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757389.761409:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757389.761420:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757389.761424:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757389.761511:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757389.761520:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757389.761525:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757390.593345:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757390.593357:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757390.593361:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757390.593370:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757390.593377:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757390.593383:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757390.593388:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757390.593393:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757390.593399:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757390.785425:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757390.785436:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757390.785444:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757390.785446:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757390.785583:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757390.785591:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757390.785599:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757391.617385:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757391.617398:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757391.617401:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757391.617409:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757391.617415:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757391.617420:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757391.617424:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757391.617428:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757391.617432:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757391.809328:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757391.809337:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757391.809345:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757391.809351:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757391.809355:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757391.809359:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757391.809363:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757391.809366:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757391.809370:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757391.809376:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757391.809423:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757391.809427:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757391.809431:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757392.641364:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757392.641378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757392.641381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757392.641390:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757392.641395:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757392.641401:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757392.641405:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757392.641410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757392.641414:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757392.833353:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757392.833359:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757392.833364:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757392.833365:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757392.833466:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757392.833474:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757392.833481:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757393.665319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757393.665334:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757393.665337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757393.665346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757393.665352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757393.665358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757393.665362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757393.665366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757393.665370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757393.857390:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757393.857404:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757393.857408:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757393.857413:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757393.857422:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757393.857426:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757393.857429:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757393.857432:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757393.857439:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757393.857441:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757393.857551:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757393.857557:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757393.857562:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757394.113446:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757394.113463:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757394.113468:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757394.113474:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757394.113483:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757394.113495:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10448 00000100:00080000:3.0:1787757394.113513:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757394.113519:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757394.113522:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757394.113529:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757394.113534:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757394.113538:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757394.113542:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757394.113545:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757394.113547:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757394.113554:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757394.113557:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757394.113563:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757394.113568:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757394.113573:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113576:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757394.113579:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757394.113599:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757394.113601:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113606:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757394.113611:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757394.113618:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113640:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757394.113654:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659643520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757394.113669:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757394.113675:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113693:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757394.113698:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757394.113703:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757394.113705:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757394.113708:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757394.113712:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757394.113715:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10448 00000100:00080000:3.0:1787757394.113719:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757394.113722:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757394.113724:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757394.113727:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757394.113730:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757394.113733:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757394.113737:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757394.113740:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757394.113742:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757394.113746:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757394.113751:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757394.113753:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757394.113762:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757394.113764:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757394.113766:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113773:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757394.113774:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787757394.113780:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757394.113781:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:3.0:1787757394.113790:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.113791:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757394.113798:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113800:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.113808:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757394.113813:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757394.113815:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757394.113819:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.113823:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113828:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757394.113835:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757394.113837:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659643520:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757394.113843:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.113849:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757394.113858:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.113859:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757394.113866:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.113872:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787757394.113880:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659643648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757394.113883:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113903:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757394.113907:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.113912:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757394.113916:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757394.113920:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757394.113923:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757394.113925:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757394.113929:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757394.113931:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757394.113938:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00080000:3.0:1787757394.113940:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10448 00000100:00000001:1.0:1787757394.113952:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757394.113953:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757394.113961:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757394.113963:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757394.113967:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757394.113969:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757394.113972:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757394.113975:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757394.113979:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757394.113981:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757394.113986:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757394.113989:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757394.113991:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757394.113996:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757394.113998:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.114000:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757394.114004:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757394.114008:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.114011:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757394.114019:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757394.114021:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.114023:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757394.114025:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757394.114027:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.114036:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757394.114043:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659643776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757394.114053:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757394.114056:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.114059:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114063:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757394.114064:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757394.114076:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757394.114079:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757394.114083:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305121985 00000100:00080000:3.0:1787757394.114083:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757394.114091:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757394.114097:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10448 00000100:00080000:3.0:1787757394.114101:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757394.114103:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757394.114105:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757394.114109:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757394.114111:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757394.114114:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787757394.114115:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757394.114123:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757394.114129:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757394.114131:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757394.114135:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757394.114138:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757394.114141:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757394.114145:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757394.114148:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114152:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305121985 00000100:00000001:3.0:1787757394.114153:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757394.114158:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757394.114164:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114166:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757394.114167:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757394.114182:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757394.114235:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757394.114269:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114275:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757394.114284:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114285:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757394.114297:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757394.114303:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757394.114305:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787757394.114309:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114311:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757394.114320:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757394.114323:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757394.114327:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114328:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757394.114333:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757394.114338:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.114343:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:3.0:1787757394.114347:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659643904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757394.114348:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757394.114359:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757394.114375:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643648:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757394.114387:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757394.114388:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757394.114397:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757394.114398:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.114402:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757394.114408:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.114409:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757394.114420:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114434:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757394.114436:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757394.114444:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757394.114473:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114484:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305121294 00000100:00000001:1.0:1787757394.114491:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305121985 00000100:00000001:1.0:1787757394.114496:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305121294 00000100:00000001:1.0:1787757394.114500:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305121985 00000100:00000001:1.0:1787757394.114504:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757394.114509:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114514:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757394.114517:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757394.114519:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.114524:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757394.114528:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757394.114532:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757394.114540:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757394.114548:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757394.114552:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757394.114555:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.114558:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757394.114565:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114575:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757394.114577:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757394.114583:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757394.114600:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114604:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305121291 00000100:00000001:1.0:1787757394.114608:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305121985 00000100:00000001:1.0:1787757394.114611:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305121291 00000100:00000001:1.0:1787757394.114614:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305121985 00000100:00000001:1.0:1787757394.114617:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757394.114619:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114623:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114625:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757394.114630:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757394.114632:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757394.114635:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757394.114636:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757394.114638:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757394.114640:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757394.114641:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.114644:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757394.114647:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757394.114649:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757394.114654:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643904:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757394.114658:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757394.114660:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757394.114662:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757394.114664:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757394.114667:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114672:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757394.114674:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:1.0:1787757394.114679:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757394.114688:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114692:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305121291 00000100:00000001:1.0:1787757394.114695:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305121985 00000100:00000001:1.0:1787757394.114698:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305121291 00000100:00000001:1.0:1787757394.114701:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305121985 00000100:00000001:1.0:1787757394.114703:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757394.114705:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114709:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114711:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757394.114714:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757394.114716:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757394.114718:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757394.114720:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757394.114722:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.114724:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757394.689329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757394.689343:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757394.689346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757394.689353:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757394.689359:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757394.689365:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757394.689369:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757394.689373:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757394.689377:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757394.881351:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757394.881363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757394.881373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757394.881376:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757394.881428:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757394.881435:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757394.881442:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757395.713314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757395.713326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757395.713329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757395.713338:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757395.713344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757395.713349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757395.713353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757395.713357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757395.713361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757395.905347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757395.905355:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757395.905362:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757395.905364:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757395.905392:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757395.905395:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757395.905399:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757396.737319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757396.737333:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757396.737336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757396.737344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757396.737350:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757396.737356:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757396.737359:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757396.737364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757396.737368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757396.929356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757396.929366:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757396.929375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757396.929377:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757396.929476:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757396.929485:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757396.929492:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787757397.185592:0:2358:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1787757397.185681:0:2358:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787757397.185712:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757397.185735:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757397.185767:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757397.185771:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757397.185774:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757397.185787:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757397.185793:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757397.185797:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757397.185798:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757397.185800:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757397.185806:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757397.185811:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757397.185814:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757397.185816:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757397.185818:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757397.185837:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757397.185843:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757397.185847:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757397.185849:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757397.185850:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757397.185864:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757397.185874:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757397.185880:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757397.185901:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757394/real 1787757397] req@ffff8ffb45c31c00 x1874588659643904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757397.185959:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757397.185965:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757397.185978:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757397.185983:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757397.185987:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.185993:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757397.185997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757397.186008:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659643904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757397.186033:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186040:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757397.186047:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186051:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757397.186053:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186058:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757397.186104:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757397.186112:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757397.186132:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757397.186139:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757397.186147:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643904:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757397.186172:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757397.186254:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757394/real 1787757397] req@ffff8ffb45c33b80 x1874588659643648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757397.186276:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757397.186279:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757397.186288:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757397.186291:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757397.186294:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186298:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757397.186303:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757397.186312:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659643648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757397.186327:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186332:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757397.186337:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186340:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757397.186342:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186345:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757397.186372:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757397.186379:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757397.186387:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757397.186392:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757397.186399:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643648:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757397.186407:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757397.186416:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757394/real 1787757397] req@ffff8ffb45c30a80 x1874588659643776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757397.186434:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757397.186437:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757397.186446:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757397.186450:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757397.186453:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186458:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757397.186461:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757397.186480:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659643776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757397.186494:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186499:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757397.186512:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186517:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757397.186520:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186523:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757397.186548:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757397.186553:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757397.186560:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757397.186564:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757397.186571:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659643776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757397.186578:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757397.186586:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757394/real 1787757397] req@ffff8ffb45c30e00 x1874588659643520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757397.186603:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757397.186606:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757397.186613:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757397.186616:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757397.186618:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186622:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757397.186624:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757397.186633:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659643520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757397.186647:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186651:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757397.186656:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186659:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757397.186662:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186665:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757397.186668:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757397.186673:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757397.186680:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757397.186685:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757397.186693:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659643520:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757397.186701:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186706:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757397.186712:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757397.186719:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757397.186730:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757397.186736:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757397.186741:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757397.186748:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757397.186753:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757397.186759:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757397.186761:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186766:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757397.186769:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757397.186772:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757397.186778:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757397.186781:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757397.186784:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757397.186788:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757397.186791:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757397.186797:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757397.186801:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186807:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757397.186811:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757397.186813:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757397.186819:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757397.186822:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757397.186825:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757397.186829:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757397.186831:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757397.186835:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757397.186837:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186840:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757397.186843:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757397.186845:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757397.186849:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757397.186851:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757397.186853:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757397.186857:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757397.186859:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757397.186862:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757397.186865:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186870:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.186885:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757397.186889:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757397.186895:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757397.186898:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757397.761375:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757397.761388:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757397.761392:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757397.761400:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757397.761406:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757397.761411:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.761415:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757397.761418:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757397.761422:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757397.953359:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757397.953371:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757397.953380:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757397.953383:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757397.953495:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757397.953503:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757397.953509:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757398.785342:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757398.785354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757398.785358:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757398.785365:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757398.785371:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757398.785376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757398.785380:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757398.785384:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757398.785389:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757398.977296:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757398.977305:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757398.977313:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757398.977316:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757398.977402:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757398.977408:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757398.977413:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757399.233302:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757399.233320:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757399.233324:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757399.233331:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757399.233338:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757399.233349:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10453 00000100:00080000:3.0:1787757399.233365:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757399.233372:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757399.233387:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757399.233395:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757399.233400:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757399.233406:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757399.233411:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757399.233415:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757399.233417:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757399.233427:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757399.233432:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757399.233438:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757399.233444:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757399.233450:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233453:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757399.233457:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757399.233468:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757399.233471:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233475:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757399.233478:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757399.233484:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233497:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757399.233509:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659644032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757399.233524:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757399.233530:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233552:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757399.233558:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757399.233563:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757399.233565:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757399.233567:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757399.233571:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757399.233574:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10453 00000100:00080000:3.0:1787757399.233580:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757399.233582:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757399.233584:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757399.233587:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757399.233589:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757399.233592:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757399.233595:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757399.233597:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757399.233599:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757399.233603:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757399.233607:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757399.233610:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757399.233616:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757399.233619:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233622:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757399.233629:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757399.233635:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233638:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757399.233642:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757399.233645:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233647:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757399.233649:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757399.233651:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233657:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.233666:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:3.0:1787757399.233667:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659644160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757399.233686:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757399.233702:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757399.233705:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757399.233712:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.233716:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757399.233720:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.233726:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757399.233727:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757399.233746:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757399.233749:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:1.0:1787757399.233752:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757399.233753:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757399.233767:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757399.233770:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757399.233778:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757399.233789:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10453 00000100:00100000:1.0:1787757399.233799:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659644032:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757399.233804:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757399.233818:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757399.233818:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757399.233829:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757399.233835:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787757399.233837:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757399.233844:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757399.233846:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757399.233851:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787757399.233857:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787757399.233864:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757399.233872:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757399.233875:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757399.233878:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757399.233890:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757399.233899:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757399.233905:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757399.233912:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757399.233917:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233920:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757399.233924:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00100000:3.0:1787757399.233928:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757399.233933:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000008:00000001:3.0:1787757399.233941:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233950:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.233951:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757399.233969:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757399.233977:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.233984:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757399.233986:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757399.233992:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.234006:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757399.234018:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659644288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757399.234035:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757399.234040:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.234046:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757399.234059:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757399.234059:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757399.234077:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757399.234080:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757399.234085:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757399.234092:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757399.234099:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757399.234101:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10453 00000100:00000001:1.0:1787757399.234116:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757399.234118:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757399.234128:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.234129:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757399.234136:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757399.234140:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757399.234271:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234283:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305127106 00000020:00000001:3.0:1787757399.234288:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757399.234295:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757399.234300:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757399.234307:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757399.234309:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757399.234319:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757399.234326:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757399.234331:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305127106 00000100:00080000:3.0:1787757399.234344:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757399.234351:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757399.234355:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234365:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757399.234366:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757399.234377:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757399.234380:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757399.234394:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234396:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757399.234406:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757399.234412:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757399.234418:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757399/real 1787757399] req@ffff8ffb45c30000 x1874588659644032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:3.0:1787757399.234423:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.234431:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757399.234441:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757399.234448:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.234462:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000400:00000001:1.0:1787757399.234481:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757399.234486:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757399.234503:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757399.234509:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757399.234512:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.234518:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757399.234521:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757399.234533:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659644032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757399.234551:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.234558:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757399.234566:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.234571:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757399.234574:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.234580:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757399.234587:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757399.234595:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757399.234604:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757399.234609:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757399.234618:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659644032:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757399.234628:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.234632:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757399.234635:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757399.234640:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757399.234650:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757399.234656:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757399.234662:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757399.234671:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757399.234675:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757399.234688:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757399.234691:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757399.234693:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757399.234699:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757399.234706:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234721:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757399.234727:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234730:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234742:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757399.234753:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:3.0:1787757399.234755:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659644416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757399.234760:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757399.234767:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757399.234773:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757399.234783:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757399.234791:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757399.234799:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757399.234803:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757399.234811:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757399.234825:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644160:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757399.234833:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234843:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757399.234852:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757399.234855:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757399.234859:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757399.234867:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234882:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757399.234886:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757399.234896:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757399.234929:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757399.234940:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757399.234947:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757399.234950:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234953:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757399.234959:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234967:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305126414 00000100:00000001:1.0:1787757399.234974:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305127106 00000100:00000001:1.0:1787757399.234978:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305126414 00000100:00000001:1.0:1787757399.234983:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305127106 00000100:00000001:1.0:1787757399.234987:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757399.234991:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.234996:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757399.235006:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757399/real 1787757399] req@ffff8ffb45c30700 x1874588659644160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757399.235023:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757399.235025:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757399.235031:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757399.235034:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757399.235038:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235041:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757399.235044:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757399.235053:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659644160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757399.235068:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235072:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757399.235078:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235082:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757399.235084:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235088:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757399.235119:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757399.235126:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757399.235134:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757399.235139:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757399.235171:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644160:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757399.235180:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757399.235232:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757399.235237:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757399.235243:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757399.235246:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757399.235250:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757399.235257:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757399.235263:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757399.235267:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757399.235271:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757399.235275:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757399.235282:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235294:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757399.235297:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787757399.235305:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757399.235333:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757399.235342:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757399.235348:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757399.235351:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235354:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757399.235360:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235368:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305126412 00000100:00000001:1.0:1787757399.235373:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305127106 00000100:00000001:1.0:1787757399.235377:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305126412 00000100:00000001:1.0:1787757399.235381:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305127106 00000100:00000001:1.0:1787757399.235385:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757399.235388:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235393:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757399.235401:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757399/real 1787757399] req@ffff8ffb45c32a00 x1874588659644288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757399.235418:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757399.235420:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757399.235426:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757399.235429:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757399.235432:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235436:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757399.235438:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757399.235447:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659644288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757399.235461:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235465:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757399.235470:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235474:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757399.235476:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235480:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757399.235505:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757399.235511:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757399.235519:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757399.235524:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757399.235531:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757399.235536:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757399.235541:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757399.235544:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757399.235548:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757399.235550:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757399.235552:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757399.235556:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757399.235558:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757399.235561:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757399.235563:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235565:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757399.235567:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757399.235569:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757399.235572:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757399.235574:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757399.235577:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757399.235581:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757399.235583:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757399.235586:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757399.235589:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235593:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235601:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757399.235604:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235608:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757399.235612:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757399.235615:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757399.235618:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757399.235621:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757399.235626:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757399.235629:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757399.235633:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757399.235640:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757399.235647:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757399.235651:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757399.235655:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757399.235658:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757399.235665:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235676:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757399.235679:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757399.235687:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757399.235714:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757399.235721:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757399.235725:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757399.235727:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235729:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757399.235734:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235739:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305126412 00000100:00000001:1.0:1787757399.235743:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305127106 00000100:00000001:1.0:1787757399.235746:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305126412 00000100:00000001:1.0:1787757399.235748:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305127106 00000100:00000001:1.0:1787757399.235751:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757399.235754:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235756:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757399.235762:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757399/real 1787757399] req@ffff8ffb45c33800 x1874588659644416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757399.235772:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757399.235774:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757399.235777:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757399.235779:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757399.235781:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235783:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757399.235785:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757399.235789:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659644416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757399.235799:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235801:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757399.235805:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235807:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757399.235808:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235810:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757399.235830:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757399.235833:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757399.235837:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757399.235839:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757399.235843:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757399.235848:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235851:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757399.235853:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757399.235855:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757399.235858:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757399.235860:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757399.235861:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757399.235864:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757399.235866:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757399.235869:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757399.235870:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235873:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.235877:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757399.235879:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.235881:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757399.235884:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757399.809316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757399.809328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757399.809332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757399.809340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757399.809345:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757399.809351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.809368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757399.809372:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757399.809376:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757400.001399:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757400.001412:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757400.001421:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757400.001423:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757400.001456:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757400.001460:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757400.001464:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757400.833341:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757400.833354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757400.833358:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757400.833367:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757400.833372:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757400.833378:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757400.833382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757400.833386:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757400.833390:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757401.025394:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757401.025405:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757401.025413:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757401.025415:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757401.025543:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757401.025552:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757401.025558:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757401.857318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757401.857327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757401.857330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757401.857336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757401.857341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757401.857346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757401.857350:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757401.857354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757401.857357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757402.049346:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757402.049354:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757402.049362:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757402.049367:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757402.049370:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757402.049374:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757402.049378:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757402.049381:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757402.049384:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757402.049387:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757402.049474:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757402.049484:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757402.049491:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757402.881291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757402.881299:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757402.881301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757402.881307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757402.881311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757402.881314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757402.881318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757402.881320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757402.881323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757403.073370:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757403.073382:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757403.073390:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757403.073392:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757403.073495:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757403.073516:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757403.073540:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757403.905277:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757403.905285:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757403.905288:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757403.905294:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757403.905299:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757403.905304:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757403.905307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757403.905310:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757403.905314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757404.097285:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757404.097298:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757404.097302:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757404.097306:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757404.097313:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757404.097317:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757404.097321:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757404.097325:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757404.097331:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757404.097334:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757404.097362:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757404.097365:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757404.097368:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757404.353336:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757404.353350:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757404.353353:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757404.353358:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757404.353364:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757404.353373:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10459 00000100:00080000:1.0:1787757404.353388:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757404.353392:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757404.353395:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757404.353400:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757404.353404:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757404.353408:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757404.353412:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757404.353415:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757404.353418:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757404.353424:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757404.353428:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.353431:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757404.353436:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757404.353440:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353443:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757404.353446:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.353459:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757404.353461:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353465:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757404.353469:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757404.353474:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353488:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757404.353500:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659644544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757404.353515:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757404.353520:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353548:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757404.353553:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757404.353557:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757404.353560:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757404.353563:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757404.353566:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757404.353569:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10459 00000100:00080000:1.0:1787757404.353574:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757404.353577:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757404.353579:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757404.353582:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757404.353584:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757404.353587:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757404.353590:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757404.353594:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757404.353596:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757404.353600:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757404.353603:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.353605:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757404.353612:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757404.353614:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353616:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757404.353622:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757404.353627:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353630:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.353636:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757404.353638:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353641:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757404.353643:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757404.353646:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353653:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757404.353660:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659644672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757404.353671:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757404.353675:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353678:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757404.353682:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757404.353686:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787757404.353688:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757404.353697:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757404.353702:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757404.353703:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757404.353716:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757404.353716:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757404.353733:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:1.0:1787757404.353738:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10459 00000100:00000001:0.0:1787757404.353740:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757404.353749:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787757404.353758:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757404.353758:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757404.353778:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757404.353779:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787757404.353800:0:87148:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:0.0:1787757404.353802:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757404.353810:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757404.353814:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757404.353822:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00100000:0.0:1787757404.353825:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659644544:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757404.353828:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757404.353837:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757404.353846:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757404.353852:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787757404.353858:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757404.353860:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.353867:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757404.353872:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.353873:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757404.353882:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353891:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757404.353894:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00100000:1.0:1787757404.353897:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757404.353904:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353906:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.353911:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757404.353914:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757404.353917:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353918:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757404.353922:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757404.353933:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.353940:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757404.353949:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659644800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757404.353960:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757404.353963:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.353965:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757404.353970:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.353973:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00080000:1.0:1787757404.353977:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757404.353990:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787757404.354001:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757404.354005:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757404.354007:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757404.354011:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757404.354016:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10459 00000100:00080000:1.0:1787757404.354046:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757404.354049:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757404.354051:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757404.354077:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757404.354079:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757404.354082:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757404.354085:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757404.354088:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757404.354090:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757404.354094:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757404.354097:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.354099:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757404.354103:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757404.354106:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.354110:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757404.354112:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757404.354120:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:1.0:1787757404.354121:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.354129:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757404.354138:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757404.354141:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.354144:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757404.354147:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757404.354149:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787757404.354150:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354161:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757404.354162:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354169:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354179:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757404.354264:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757404.354267:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659644928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757404.354273:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305132226 00000100:00000001:1.0:1787757404.354297:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757404.354301:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354306:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757404.354312:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354322:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305132226 00000100:00000001:0.0:1787757404.354334:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757404.354356:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757404.354361:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354365:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757404.354376:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757404/real 1787757404] req@ffff8ffb4a537800 x1874588659644544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757404.354395:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757404.354399:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757404.354407:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757404.354412:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757404.354414:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354418:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757404.354421:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757404.354428:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659644544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757404.354439:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354444:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757404.354449:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354451:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757404.354453:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354457:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757404.354460:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757404.354465:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757404.354471:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757404.354474:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757404.354479:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659644544:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757404.354486:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757404.354489:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757404.354491:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757404.354494:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757404.354497:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757404.354500:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757404.354504:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644672:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757404.354509:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757404.354512:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757404.354514:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757404.354516:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757404.354520:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354527:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757404.354529:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:0.0:1787757404.354537:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757404.354554:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757404.354561:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757404.354564:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757404.354566:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354568:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757404.354571:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354577:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305131534 00000100:00000001:0.0:1787757404.354581:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305132226 00000100:00000001:0.0:1787757404.354584:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305131534 00000100:00000001:0.0:1787757404.354587:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305132226 00000100:00000001:0.0:1787757404.354591:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757404.354594:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354596:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757404.354601:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757404/real 1787757404] req@ffff8ffb4a536300 x1874588659644672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757404.354612:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757404.354613:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757404.354616:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757404.354617:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757404.354619:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354621:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757404.354623:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757404.354627:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659644672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757404.354637:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354640:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757404.354643:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354645:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757404.354646:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354648:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757404.354671:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757404.354674:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757404.354677:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757404.354680:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757404.354684:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644672:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757404.354689:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354692:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757404.354694:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757404.354699:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757404.354705:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757404.354708:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757404.354712:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757404.354720:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757404.354723:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757404.354727:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757404.354729:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354731:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757404.354733:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757404.354734:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757404.354737:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757404.354739:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757404.354741:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757404.354743:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757404.354744:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757404.354746:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757404.354748:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354751:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354761:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757404.354763:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354767:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757404.354769:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757404.354771:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757404.354773:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757404.354775:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757404.354778:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757404.354780:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757404.354783:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757404.354787:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757404.354791:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757404.354793:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757404.354795:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757404.354797:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757404.354800:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354806:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757404.354808:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787757404.354813:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757404.354825:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757404.354830:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757404.354832:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757404.354834:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354837:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757404.354841:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354845:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305131531 00000100:00000001:0.0:1787757404.354848:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305132226 00000100:00000001:0.0:1787757404.354851:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305131531 00000100:00000001:0.0:1787757404.354854:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305132226 00000100:00000001:0.0:1787757404.354857:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757404.354859:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354861:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757404.354866:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757404/real 1787757404] req@ffff8ffb4a537b80 x1874588659644800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757404.354875:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757404.354877:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757404.354880:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757404.354882:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757404.354884:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354886:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757404.354888:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757404.354892:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659644800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757404.354901:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354903:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757404.354906:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354908:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757404.354909:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.354911:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757404.354930:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757404.354933:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757404.354937:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757404.354939:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757404.354944:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757404.354949:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757404.354951:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757404.354953:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757404.354956:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757404.354958:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757404.354960:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757404.354964:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757404.354967:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757404.354969:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757404.354972:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757404.354974:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757404.354977:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.354982:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757404.354984:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787757404.354989:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757404.355000:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757404.355004:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757404.355007:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757404.355009:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.355011:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757404.355014:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.355018:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305131531 00000100:00000001:0.0:1787757404.355022:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305132226 00000100:00000001:0.0:1787757404.355024:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305131531 00000100:00000001:0.0:1787757404.355028:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305132226 00000100:00000001:0.0:1787757404.355030:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757404.355033:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.355035:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757404.355039:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757404/real 1787757404] req@ffff8ffb4a534700 x1874588659644928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757404.355084:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757404.355086:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757404.355090:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757404.355092:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757404.355094:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.355096:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757404.355098:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757404.355103:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659644928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757404.355113:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.355116:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757404.355118:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.355120:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757404.355122:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.355124:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757404.355143:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757404.355146:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757404.355149:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757404.355152:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757404.355155:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659644928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757404.355160:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.355163:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757404.355165:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757404.355167:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757404.355170:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757404.355172:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757404.355173:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757404.355177:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757404.355179:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757404.355215:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757404.355218:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.355221:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757404.355223:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757404.355224:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757404.355227:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757404.355228:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757404.355230:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757404.355232:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757404.355234:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757404.355236:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757404.355237:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.355239:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757404.355243:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757404.355245:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757404.355247:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757404.355249:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757404.929291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757404.929300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757404.929303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757404.929311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757404.929316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757404.929321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757404.929324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757404.929327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757404.929330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757405.121375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757405.121386:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757405.121398:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757405.121403:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757405.121472:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757405.121481:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757405.121489:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757405.953279:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757405.953289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757405.953291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757405.953299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757405.953303:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757405.953308:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757405.953312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757405.953315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757405.953319:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757406.145321:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757406.145332:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757406.145345:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757406.145348:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757406.145457:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757406.145463:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757406.145468:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757406.977326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757406.977338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757406.977342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757406.977350:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757406.977355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757406.977361:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757406.977365:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757406.977368:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757406.977372:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757407.169340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757407.169351:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757407.169360:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757407.169362:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757407.169476:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757407.169483:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757407.169488:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757408.001316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757408.001326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757408.001329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757408.001336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757408.001340:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757408.001345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757408.001348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757408.001352:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757408.001356:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757408.193335:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757408.193343:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757408.193351:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757408.193353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757408.193387:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757408.193391:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757408.193395:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757409.025320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757409.025329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757409.025332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757409.025339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757409.025343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757409.025348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757409.025352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757409.025356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757409.025360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757409.217316:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757409.217325:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757409.217333:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757409.217336:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757409.217444:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757409.217453:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757409.217460:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757409.473421:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757409.473438:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757409.473442:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757409.473448:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757409.473455:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757409.473467:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10464 00000100:00080000:2.0:1787757409.473485:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757409.473490:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757409.473493:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757409.473500:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757409.473504:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757409.473509:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757409.473512:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757409.473515:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757409.473517:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757409.473524:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757409.473527:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757409.473531:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757409.473536:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757409.473541:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473544:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757409.473547:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757409.473567:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757409.473569:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473575:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757409.473579:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757409.473588:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473612:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757409.473632:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659645056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757409.473655:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757409.473665:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473702:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757409.473709:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757409.473715:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757409.473718:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757409.473720:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757409.473725:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757409.473729:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10464 00000100:00080000:2.0:1787757409.473735:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757409.473738:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757409.473740:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757409.473744:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757409.473745:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757409.473748:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757409.473751:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757409.473754:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757409.473756:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757409.473760:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757409.473763:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757409.473765:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757409.473773:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757409.473776:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473778:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757409.473784:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757409.473789:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473792:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757409.473799:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757409.473802:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473805:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757409.473808:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757409.473811:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473820:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757409.473827:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659645184/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757409.473839:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757409.473841:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757409.473847:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473860:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.473861:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:2.0:1787757409.473865:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757409.473871:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:2.0:1787757409.473872:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757409.473882:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757409.473884:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:2.0:1787757409.473886:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757409.473894:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473894:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757409.473904:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10464 00000100:00080000:2.0:1787757409.473910:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787757409.473912:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757409.473913:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757409.473923:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757409.473926:0:87148:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:0.0:1787757409.473927:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787757409.473929:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757409.473933:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757409.473946:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757409.473948:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.473950:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757409.473954:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757409.473962:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757409.473995:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757409.474002:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659645056:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757409.474003:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757409.474008:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757409.474026:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474030:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757409.474032:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757409.474038:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757409.474040:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757409.474053:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757409.474054:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.474076:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757409.474078:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757409.474097:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757409.474099:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757409.474102:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.474111:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757409.474114:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757409.474119:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474129:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757409.474129:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474139:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00100000:2.0:1787757409.474152:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659645312/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757409.474153:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757409.474170:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757409.474174:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.474261:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757409.474273:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787757409.474278:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757409.474285:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757409.474289:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757409.474293:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757409.474295:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757409.474307:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757409.474309:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757409.474314:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757409.474315:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10464 00000100:00000001:0.0:1787757409.474326:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:2.0:1787757409.474333:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757409.474338:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.474340:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757409.474348:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757409.474350:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305137346 00000020:00000001:2.0:1787757409.474357:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757409.474361:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757409.474364:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757409.474371:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:2.0:1787757409.474371:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757409.474386:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757409.474389:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757409.474391:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305137346 00000100:00080000:2.0:1787757409.474399:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757409.474407:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474409:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757409.474429:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757409.474436:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757409.474440:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.474442:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757409.474444:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757409.474450:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474451:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757409.474461:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474463:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474475:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757409.474477:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757409.474489:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.474494:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757409.474501:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:0.0:1787757409.474502:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757409/real 1787757409] req@ffff8ffb47a38700 x1874588659645056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757409.474508:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757409.474529:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757409.474534:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757409.474542:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:2.0:1787757409.474543:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659645440/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757409.474552:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757409.474562:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757409.474563:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757409.474578:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757409.474580:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474590:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757409.474593:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757409.474597:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757409.474606:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659645056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757409.474620:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474625:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757409.474629:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474632:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757409.474634:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474637:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757409.474640:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757409.474644:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757409.474652:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757409.474658:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757409.474663:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659645056:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757409.474671:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474674:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757409.474676:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757409.474679:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757409.474685:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757409.474689:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757409.474692:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757409.474697:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757409.474699:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757409.474704:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757409.474705:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474708:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474717:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757409.474719:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474722:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757409.474725:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757409.474727:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757409.474729:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757409.474732:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757409.474735:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757409.474739:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757409.474742:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757409.474746:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645184:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757409.474751:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757409.474753:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757409.474756:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757409.474758:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757409.474762:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474768:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757409.474770:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:0.0:1787757409.474776:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757409.474792:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757409.474798:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757409.474801:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757409.474803:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474805:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757409.474809:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474814:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305136654 00000100:00000001:0.0:1787757409.474818:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305137346 00000100:00000001:0.0:1787757409.474821:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305136654 00000100:00000001:0.0:1787757409.474823:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305137346 00000100:00000001:0.0:1787757409.474826:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757409.474829:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474831:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757409.474836:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757409/real 1787757409] req@ffff8ffb47a3a300 x1874588659645184/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757409.474846:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757409.474847:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757409.474850:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757409.474852:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757409.474854:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474857:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757409.474859:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757409.474863:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659645184/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757409.474872:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474875:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757409.474878:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474880:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757409.474881:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.474883:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757409.474900:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757409.474903:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757409.474907:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757409.474910:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757409.474913:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645184:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757409.474918:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757409.474920:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757409.474922:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757409.474925:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757409.474927:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757409.474930:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757409.474933:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757409.474937:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757409.474939:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757409.474941:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757409.474943:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757409.474946:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.474952:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757409.474954:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757409.474958:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757409.474998:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757409.475004:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757409.475007:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757409.475009:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.475011:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757409.475015:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.475019:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305136651 00000100:00000001:0.0:1787757409.475022:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305137346 00000100:00000001:0.0:1787757409.475025:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305136651 00000100:00000001:0.0:1787757409.475028:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305137346 00000100:00000001:0.0:1787757409.475031:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757409.475033:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.475035:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757409.475041:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757409/real 1787757409] req@ffff8ffb47a39180 x1874588659645312/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757409.475050:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757409.475051:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757409.475054:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757409.475056:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757409.475057:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475059:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757409.475061:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757409.475066:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659645312/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757409.475074:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475077:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757409.475079:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475081:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757409.475083:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475085:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757409.475098:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757409.475101:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757409.475106:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757409.475108:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757409.475113:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757409.475117:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757409.475119:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757409.475121:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757409.475124:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757409.475126:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757409.475128:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757409.475131:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757409.475135:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757409.475137:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757409.475139:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757409.475142:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757409.475145:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.475151:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757409.475153:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757409.475157:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757409.475169:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757409.475174:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757409.475176:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757409.475178:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.475210:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757409.475215:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.475220:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305136652 00000100:00000001:0.0:1787757409.475223:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305137346 00000100:00000001:0.0:1787757409.475226:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305136652 00000100:00000001:0.0:1787757409.475229:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305137346 00000100:00000001:0.0:1787757409.475232:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757409.475234:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.475236:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757409.475243:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757409/real 1787757409] req@ffff8ffb47a3b480 x1874588659645440/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757409.475253:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757409.475255:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757409.475258:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757409.475260:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757409.475262:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475264:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757409.475266:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757409.475270:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659645440/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757409.475279:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475281:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757409.475284:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475286:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757409.475287:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475289:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757409.475304:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757409.475307:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757409.475310:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757409.475312:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757409.475316:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757409.475320:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475322:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757409.475324:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757409.475326:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757409.475329:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757409.475331:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757409.475333:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757409.475335:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757409.475337:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757409.475341:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757409.475343:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475345:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757409.475347:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757409.475348:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757409.475351:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757409.475353:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757409.475354:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757409.475356:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757409.475357:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757409.475360:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757409.475362:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475364:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757409.475365:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757409.475366:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757409.475369:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757409.475370:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757409.475371:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757409.475373:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757409.475374:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757409.475377:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757409.475378:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475380:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757409.475384:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757409.475385:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757409.475388:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757409.475390:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757410.049275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757410.049283:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757410.049285:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757410.049290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757410.049294:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757410.049298:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757410.049300:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757410.049302:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757410.049304:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757410.241262:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757410.241268:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757410.241273:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757410.241275:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757410.241307:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757410.241311:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757410.241315:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757411.073321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757411.073333:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757411.073336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757411.073344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757411.073349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757411.073353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757411.073357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757411.073360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757411.073364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757411.265389:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757411.265401:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757411.265410:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757411.265416:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757411.265420:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757411.265425:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757411.265430:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757411.265433:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757411.265436:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757411.265439:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757411.265536:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757411.265548:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757411.265557:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757412.097330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757412.097340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757412.097343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757412.097349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757412.097354:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757412.097359:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757412.097363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757412.097366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757412.097370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757412.289393:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757412.289409:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757412.289422:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757412.289425:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757412.289472:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757412.289476:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757412.289481:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757413.121307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757413.121318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757413.121322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757413.121331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757413.121337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757413.121343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757413.121348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757413.121354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757413.121360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757413.313380:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757413.313400:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757413.313407:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757413.313415:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757413.313426:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757413.313431:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757413.313435:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757413.313438:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757413.313445:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757413.313448:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757413.313532:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757413.313540:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757413.313547:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757414.145373:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757414.145385:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757414.145388:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757414.145396:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757414.145402:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757414.145408:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757414.145412:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757414.145416:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757414.145420:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757414.337334:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757414.337347:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757414.337356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757414.337358:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757414.337413:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757414.337419:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757414.337423:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757414.593324:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757414.593337:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757414.593342:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757414.593347:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757414.593353:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757414.593362:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10469 00000100:00080000:2.0:1787757414.593376:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757414.593382:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757414.593386:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757414.593395:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757414.593401:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757414.593407:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757414.593413:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757414.593419:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757414.593423:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757414.593433:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757414.593440:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757414.593447:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757414.593454:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757414.593460:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593465:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757414.593469:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757414.593485:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757414.593489:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593495:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757414.593501:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757414.593507:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593523:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757414.593541:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659645568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757414.593559:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757414.593567:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593600:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757414.593608:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757414.593615:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757414.593618:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757414.593622:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757414.593629:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757414.593636:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10469 00000100:00080000:2.0:1787757414.593646:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757414.593651:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757414.593656:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757414.593661:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757414.593664:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757414.593665:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:2.0:1787757414.593668:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757414.593674:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787757414.593676:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757414.593682:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757414.593683:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757414.593690:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757414.593691:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757414.593696:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757414.593708:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:2.0:1787757414.593710:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757414.593713:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757414.593722:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593726:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757414.593736:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659645568:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757414.593738:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757414.593748:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757414.593752:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757414.593754:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757414.593761:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757414.593764:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757414.593766:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757414.593776:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757414.593778:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:2.0:1787757414.593787:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757414.593796:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593802:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.593803:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757414.593809:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:1.0:1787757414.593817:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787757414.593819:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757414.593827:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593832:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757414.593836:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757414.593841:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593855:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757414.593870:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659645696/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757414.593871:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757414.593916:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757414.593921:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757414.593925:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757414.593933:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.593933:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.593946:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757414.593952:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.593956:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757414.593964:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757414.593966:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305142465 00000100:00080000:2.0:1787757414.593981:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757414.593985:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757414.593986:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757414.593995:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757414.594000:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305142465 00000100:00000001:2.0:1787757414.594007:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757414.594009:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:2.0:1787757414.594021:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10469 00000100:00000001:1.0:1787757414.594029:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787757414.594034:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757414.594036:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594048:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757414.594049:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787757414.594060:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757414.594065:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757414/real 1787757414] req@ffff8ffb47a38000 x1874588659645568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787757414.594071:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757414.594078:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757414.594087:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757414.594101:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757414.594113:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:1.0:1787757414.594114:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757414.594120:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:1.0:1787757414.594121:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757414.594130:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:2.0:1787757414.594138:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757414.594140:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757414.594148:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757414.594151:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594154:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757414.594163:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757414.594164:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:1.0:1787757414.594174:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659645568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757414.594176:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757414.594225:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757414.594236:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594237:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757414.594250:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757414.594251:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.594260:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757414.594263:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:2.0:1787757414.594263:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757414.594272:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000008:00000001:2.0:1787757414.594279:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594280:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757414.594291:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:2.0:1787757414.594292:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757414.594309:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757414.594313:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757414.594319:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757414.594322:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757414.594332:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757414.594333:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:2.0:1787757414.594342:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757414.594343:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659645568:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757414.594358:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.594363:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757414.594363:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594368:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757414.594374:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757414.594381:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757414.594386:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00100000:2.0:1787757414.594388:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659645824/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000001:1.0:1787757414.594390:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757414.594397:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757414.594401:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757414.594408:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757414.594413:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.594421:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757414.594423:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757414.594431:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594435:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757414.594438:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757414.594441:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:2.0:1787757414.594444:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757414.594447:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594451:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757414.594453:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787757414.594457:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757414.594459:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757414.594464:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757414.594467:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757414.594469:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757414.594475:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757414.594478:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10469 02000000:00000001:1.0:1787757414.594480:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757414.594487:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787757414.594491:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757414.594492:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594500:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:1.0:1787757414.594503:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645696:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757414.594504:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757414.594512:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757414.594519:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:2.0:1787757414.594520:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757414.594525:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:2.0:1787757414.594527:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:1.0:1787757414.594530:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757414.594534:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:1.0:1787757414.594536:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757414.594544:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787757414.594545:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594549:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757414.594559:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787757414.594560:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757414.594566:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:2.0:1787757414.594569:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757414.594576:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757414.594577:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787757414.594583:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757414.594590:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594592:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757414.594596:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757414.594600:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594607:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757414.594608:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757414.594622:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787757414.594622:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757414.594629:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594631:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757414.594636:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757414.594638:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757414.594642:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757414.594645:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787757414.594648:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594652:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594660:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594661:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305141774 00000100:00000001:1.0:1787757414.594671:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305142465 00000100:00100000:2.0:1787757414.594678:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659645952/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757414.594679:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305141774 00000100:00000001:1.0:1787757414.594691:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305142465 00000100:00000001:1.0:1787757414.594700:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757414.594711:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757414.594714:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757414.594716:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787757414.594722:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757414.594731:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757414/real 1787757414] req@ffff8ffb47a38380 x1874588659645696/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757414.594732:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757414.594759:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757414.594762:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757414.594766:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757414.594772:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757414.594774:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.594777:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757414.594779:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757414.594785:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659645696/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757414.594794:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.594797:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757414.594800:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.594802:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757414.594803:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.594805:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757414.594820:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757414.594824:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757414.594827:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757414.594830:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757414.594834:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645696:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757414.594839:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757414.594841:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757414.594843:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757414.594846:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757414.594848:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757414.594851:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757414.594855:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645824:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757414.594858:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757414.594860:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757414.594863:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757414.594865:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757414.594870:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594900:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757414.594903:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757414.594909:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757414.594924:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757414.594930:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757414.594933:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757414.594934:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594937:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757414.594940:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594944:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305141771 00000100:00000001:1.0:1787757414.594947:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305142465 00000100:00000001:1.0:1787757414.594951:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305141771 00000100:00000001:1.0:1787757414.594954:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305142465 00000100:00000001:1.0:1787757414.594957:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757414.594960:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.594962:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757414.594967:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757414/real 1787757414] req@ffff8ffb47a38a80 x1874588659645824/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757414.594976:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757414.594977:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757414.594980:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757414.594982:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757414.594984:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.594986:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757414.594989:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757414.594994:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659645824/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757414.595003:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595006:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757414.595010:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595012:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757414.595014:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595016:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757414.595028:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757414.595031:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757414.595036:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757414.595038:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757414.595042:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645824:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757414.595046:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595049:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757414.595050:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757414.595052:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757414.595054:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757414.595056:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757414.595058:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757414.595061:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757414.595063:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757414.595066:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757414.595068:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595070:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757414.595071:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757414.595073:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757414.595076:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757414.595078:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757414.595079:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757414.595081:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757414.595082:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757414.595084:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757414.595086:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595088:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595092:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757414.595095:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.595097:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757414.595099:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757414.595101:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757414.595103:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757414.595106:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757414.595109:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757414.595111:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757414.595114:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757414.595117:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645952:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757414.595121:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757414.595123:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757414.595125:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757414.595127:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757414.595130:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.595135:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757414.595137:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757414.595142:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757414.595155:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757414.595160:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757414.595163:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757414.595165:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.595167:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757414.595170:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.595174:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305141771 00000100:00000001:1.0:1787757414.595177:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305142465 00000100:00000001:1.0:1787757414.595181:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305141771 00000100:00000001:1.0:1787757414.595221:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305142465 00000100:00000001:1.0:1787757414.595225:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757414.595229:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.595232:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757414.595238:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757414/real 1787757414] req@ffff8ffb47a3a680 x1874588659645952/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757414.595249:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757414.595252:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757414.595259:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757414.595261:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757414.595265:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595269:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757414.595271:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757414.595277:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659645952/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757414.595288:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595293:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757414.595296:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595299:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757414.595301:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595304:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757414.595318:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757414.595322:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757414.595325:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757414.595328:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757414.595331:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659645952:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757414.595337:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595339:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757414.595341:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757414.595343:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757414.595345:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757414.595347:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757414.595348:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757414.595351:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757414.595353:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757414.595356:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757414.595358:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595360:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757414.595364:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757414.595366:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757414.595369:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757414.595371:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757415.169334:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757415.169346:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757415.169349:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757415.169358:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757415.169364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757415.169370:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757415.169373:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757415.169377:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757415.169380:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757415.361319:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757415.361329:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757415.361339:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757415.361340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757415.361372:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757415.361376:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757415.361379:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757416.193268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757416.193276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757416.193279:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757416.193285:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757416.193289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757416.193294:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757416.193299:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757416.193304:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757416.193309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757416.385317:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757416.385327:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757416.385337:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757416.385340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757416.385377:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757416.385383:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757416.385389:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757417.217306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757417.217315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757417.217318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757417.217324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757417.217329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757417.217334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757417.217337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757417.217341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757417.217344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757417.409347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757417.409354:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757417.409361:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757417.409363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757417.409492:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757417.409498:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757417.409503:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757418.241416:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757418.241428:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757418.241431:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757418.241438:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757418.241444:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757418.241452:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757418.241456:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757418.241460:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757418.241463:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757418.433372:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757418.433382:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757418.433390:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757418.433392:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757418.433485:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757418.433493:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757418.433500:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757419.265256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757419.265262:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757419.265263:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757419.265267:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757419.265270:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757419.265273:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757419.265275:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757419.265277:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757419.265279:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757419.457263:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757419.457269:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757419.457273:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757419.457274:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757419.457334:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757419.457337:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757419.457340:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757419.713264:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757419.713272:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757419.713274:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757419.713277:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757419.713281:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757419.713287:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10474 00000100:00080000:3.0:1787757419.713296:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757419.713298:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757419.713300:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757419.713303:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757419.713305:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757419.713307:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757419.713309:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757419.713311:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757419.713312:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757419.713315:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757419.713317:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757419.713319:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757419.713321:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757419.713323:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713325:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757419.713327:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757419.713336:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757419.713337:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713339:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757419.713342:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757419.713345:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713355:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757419.713362:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659646080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757419.713370:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757419.713374:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713386:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757419.713389:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757419.713392:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757419.713393:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757419.713394:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757419.713396:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757419.713397:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10474 00000100:00080000:3.0:1787757419.713400:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757419.713401:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757419.713403:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757419.713405:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757419.713407:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757419.713408:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757419.713410:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757419.713412:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757419.713412:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757419.713415:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787757419.713415:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757419.713417:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757419.713421:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757419.713422:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757419.713426:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757419.713427:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757419.713430:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757419.713431:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713434:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713434:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757419.713440:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00100000:3.0:1787757419.713442:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757419.713444:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787757419.713446:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.713448:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713449:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757419.713454:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:1.0:1787757419.713455:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659646080:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757419.713456:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713460:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757419.713462:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757419.713465:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757419.713466:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.713470:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757419.713471:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.713473:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.713476:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787757419.713480:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659646208/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757419.713482:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713491:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757419.713493:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713496:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757419.713498:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757419.713500:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757419.713501:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757419.713503:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757419.713504:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757419.713505:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757419.713508:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00080000:3.0:1787757419.713509:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10474 00000100:00080000:3.0:1787757419.713513:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757419.713515:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757419.713517:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757419.713517:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757419.713522:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757419.713523:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757419.713524:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757419.713526:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757419.713527:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757419.713528:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757419.713530:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757419.713531:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757419.713532:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757419.713534:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757419.713536:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713537:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757419.713539:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757419.713540:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713541:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757419.713544:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757419.713546:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713547:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757419.713549:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757419.713550:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713555:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757419.713562:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1787757419.713568:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659646336/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757419.713575:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757419.713576:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757419.713577:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713582:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757419.713585:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713586:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713588:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757419.713591:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757419.713595:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757419.713596:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713596:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757419.713600:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757419.713603:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757419.713604:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10474 00000100:00080000:3.0:1787757419.713607:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757419.713608:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757419.713608:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757419.713610:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757419.713610:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757419.713611:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305147585 00000020:00000001:3.0:1787757419.713612:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757419.713617:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757419.713619:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757419.713620:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757419.713623:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757419.713624:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713625:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757419.713626:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757419.713631:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305147585 02000000:00000001:3.0:1787757419.713633:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757419.713637:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713638:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757419.713639:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757419.713643:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757419.713644:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787757419.713647:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713648:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713650:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713652:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787757419.713655:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757419.713658:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757419.713661:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757419/real 1787757419] req@ffff8ffb45c31500 x1874588659646080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757419.713661:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757419.713664:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757419.713666:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757419.713672:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757419.713678:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00100000:3.0:1787757419.713680:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659646464/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757419.713681:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757419.713685:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757419.713689:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757419.713690:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757419.713692:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757419.713693:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713699:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757419.713700:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713703:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757419.713710:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659646080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757419.713716:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713719:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757419.713723:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713725:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757419.713727:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713730:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757419.713732:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757419.713735:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757419.713740:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757419.713743:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757419.713748:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659646080:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757419.713753:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713756:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757419.713758:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757419.713761:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757419.713765:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757419.713767:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757419.713770:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757419.713781:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757419.713783:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757419.713811:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757419.713813:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713815:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713820:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757419.713822:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713824:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757419.713825:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757419.713826:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757419.713828:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757419.713829:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.713831:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757419.713833:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757419.713835:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757419.713838:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646208:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757419.713840:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757419.713842:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757419.713843:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.713845:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757419.713847:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713852:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757419.713854:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757419.713857:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757419.713867:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757419.713871:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757419.713873:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757419.713874:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713875:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757419.713877:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713880:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305146893 00000100:00000001:1.0:1787757419.713883:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305147585 00000100:00000001:1.0:1787757419.713885:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305146893 00000100:00000001:1.0:1787757419.713886:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305147585 00000100:00000001:1.0:1787757419.713888:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757419.713889:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713890:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757419.713893:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757419/real 1787757419] req@ffff8ffb45c31180 x1874588659646208/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757419.713899:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757419.713900:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757419.713902:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757419.713903:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757419.713904:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713905:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757419.713906:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757419.713909:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659646208/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757419.713915:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713916:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757419.713918:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713919:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757419.713920:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.713921:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757419.713931:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757419.713932:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757419.713934:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757419.713936:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757419.713938:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646208:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757419.713940:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757419.713941:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757419.713943:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.713944:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757419.713946:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757419.713947:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757419.713949:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757419.713951:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757419.713952:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757419.713952:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.713954:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757419.713955:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713958:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757419.713960:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:1.0:1787757419.713963:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757419.713969:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757419.713972:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757419.713974:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757419.713976:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713977:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757419.713979:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713981:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305146890 00000100:00000001:1.0:1787757419.713983:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305147585 00000100:00000001:1.0:1787757419.713985:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305146890 00000100:00000001:1.0:1787757419.713987:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305147585 00000100:00000001:1.0:1787757419.713988:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757419.713989:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.713990:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757419.713993:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757419/real 1787757419] req@ffff8ffb45c33480 x1874588659646336/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757419.713997:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757419.713998:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757419.713999:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757419.714000:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757419.714001:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714002:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757419.714003:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757419.714005:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659646336/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757419.714010:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714012:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757419.714013:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714014:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757419.714015:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714016:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757419.714024:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757419.714026:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757419.714028:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757419.714029:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757419.714032:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757419.714035:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757419.714036:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757419.714037:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.714038:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757419.714039:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757419.714040:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757419.714042:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646464:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757419.714044:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757419.714045:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757419.714045:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757419.714047:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757419.714048:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.714051:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757419.714051:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757419.714054:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757419.714060:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757419.714062:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757419.714064:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757419.714065:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.714066:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757419.714068:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.714070:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305146890 00000100:00000001:1.0:1787757419.714072:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305147585 00000100:00000001:1.0:1787757419.714074:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305146890 00000100:00000001:1.0:1787757419.714076:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305147585 00000100:00000001:1.0:1787757419.714077:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757419.714079:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.714080:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757419.714082:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757419/real 1787757419] req@ffff8ffb45c33800 x1874588659646464/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757419.714092:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757419.714093:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757419.714094:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757419.714095:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757419.714096:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714097:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757419.714098:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757419.714100:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659646464/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757419.714104:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714105:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757419.714106:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714107:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757419.714108:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714109:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757419.714118:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757419.714119:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757419.714121:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757419.714122:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757419.714124:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646464:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757419.714126:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714127:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757419.714128:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757419.714129:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757419.714131:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757419.714132:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757419.714133:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757419.714134:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757419.714135:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757419.714137:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757419.714138:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714139:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757419.714140:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757419.714141:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757419.714143:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757419.714144:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757419.714145:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757419.714146:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757419.714147:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757419.714148:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757419.714149:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714150:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757419.714150:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757419.714151:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757419.714152:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757419.714153:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757419.714153:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757419.714154:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757419.714155:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757419.714155:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757419.714156:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714157:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757419.714159:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757419.714160:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757419.714161:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757419.714162:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757420.289301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757420.289308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757420.289309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757420.289314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757420.289317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757420.289320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757420.289323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757420.289325:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757420.289327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757420.481309:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757420.481315:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757420.481320:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757420.481322:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757420.481344:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757420.481347:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757420.481349:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757421.313331:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757421.313340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757421.313343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757421.313349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757421.313354:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757421.313358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757421.313362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757421.313365:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757421.313368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757421.505316:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757421.505327:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757421.505337:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757421.505343:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757421.505347:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757421.505354:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757421.505359:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757421.505362:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757421.505366:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757421.505370:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757421.505480:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757421.505493:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757421.505500:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757422.337346:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757422.337359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757422.337362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757422.337370:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757422.337376:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757422.337382:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757422.337386:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757422.337390:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757422.337394:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757422.529400:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757422.529416:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757422.529432:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757422.529438:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757422.529542:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757422.529548:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757422.529554:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757423.361304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757423.361317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757423.361320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757423.361328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757423.361334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757423.361341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757423.361344:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757423.361350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757423.361355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757423.553328:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757423.553342:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757423.553347:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757423.553353:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757423.553360:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757423.553363:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757423.553367:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757423.553370:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757423.553375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757423.553377:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757423.553495:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757423.553503:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757423.553510:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757424.385300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757424.385309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757424.385325:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757424.385335:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757424.385341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757424.385347:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757424.385352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757424.385357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757424.385363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757424.577344:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757424.577363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757424.577371:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757424.577374:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757424.577407:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757424.577410:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757424.577414:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757424.833330:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757424.833342:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757424.833345:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757424.833350:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757424.833355:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757424.833364:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10479 00000100:00080000:3.0:1787757424.833378:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757424.833382:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757424.833385:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757424.833390:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757424.833393:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757424.833397:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757424.833400:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757424.833404:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757424.833406:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757424.833411:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757424.833415:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757424.833419:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757424.833423:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757424.833426:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.833429:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757424.833432:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757424.833446:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757424.833448:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.833452:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757424.833455:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757424.833460:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.833474:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757424.833486:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659646592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757424.833500:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757424.833505:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.833532:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757424.833539:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757424.833546:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757424.833548:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757424.833551:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757424.833557:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757424.833562:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10479 00000100:00080000:3.0:1787757424.833570:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757424.833574:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757424.833577:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757424.833582:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757424.833584:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757424.833588:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757424.833594:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757424.833598:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757424.833601:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757424.833608:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757424.833615:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757424.833619:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757424.833630:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757424.833634:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.833637:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757424.833644:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757424.833649:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.833651:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757424.833660:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757424.833662:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757424.833665:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.833676:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757424.833676:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757424.833689:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757424.833690:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757424.833723:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757424.833733:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757424.833735:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.833754:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757424.833761:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757424.833762:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.833770:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757424.833792:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659646592:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787757424.833795:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659646720/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757424.833815:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757424.833831:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757424.833832:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757424.833839:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.833840:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.833855:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757424.833862:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757424.833869:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757424.833872:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757424.833875:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757424.833880:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757424.833885:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757424.833892:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757424.833899:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10479 00000100:00000001:1.0:1787757424.833907:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757424.833907:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757424.833915:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757424.833915:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757424.833924:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757424.833933:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787757424.833935:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757424.833950:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757424.833956:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757424.833961:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757424.833964:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757424.833965:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757424.833972:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757424.833976:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757424.833979:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757424.833984:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757424.833987:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.833991:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757424.833995:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757424.833998:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.834001:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757424.834010:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757424.834017:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757424.834019:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757424.834029:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757424.834033:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757424.834037:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834048:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757424.834048:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834061:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00100000:3.0:1787757424.834064:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659646848/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757424.834067:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834087:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757424.834099:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.834100:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757424.834111:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834114:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305152705 00000100:00000001:3.0:1787757424.834117:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757424.834125:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757424.834131:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757424.834134:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757424.834135:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757424.834147:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757424.834151:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305152705 00000100:00000001:3.0:1787757424.834155:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757424.834161:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757424.834171:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757424.834171:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10479 00000100:00000001:1.0:1787757424.834180:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834244:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:3.0:1787757424.834245:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757424.834252:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757424.834258:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757424.834260:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757424/real 1787757424] req@ffff8ffb45c32a00 x1874588659646592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757424.834270:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757424.834275:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757424.834282:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757424.834306:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787757424.834307:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757424.834313:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757424.834316:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757424.834320:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757424.834323:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757424.834330:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787757424.834331:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757424.834341:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757424.834342:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834354:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757424.834354:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757424.834374:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757424.834377:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757424.834386:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757424.834394:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659646592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757424.834398:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757424.834405:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757424.834421:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757424.834434:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834437:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.834450:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787757424.834451:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757424.834460:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834463:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757424.834470:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757424.834471:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757424.834488:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757424.834490:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757424.834504:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:3.0:1787757424.834505:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834516:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757424.834526:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757424.834527:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757424.834539:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757424.834545:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757424.834556:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659646976/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757424.834559:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659646592:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757424.834579:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.834589:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757424.834596:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757424.834604:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757424.834616:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757424.834619:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000020:00000001:1.0:1787757424.834623:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:3.0:1787757424.834625:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757424.834629:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757424.834635:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.834639:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757424.834647:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757424.834655:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757424.834657:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.834661:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.834672:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757424.834675:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834680:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757424.834684:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757424.834687:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757424.834691:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757424.834715:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.834720:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757424.834723:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757424.834727:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757424.834733:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646720:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757424.834739:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757424.834742:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757424.834745:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.834747:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757424.834752:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834759:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757424.834761:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757424.834767:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757424.834786:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757424.834792:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757424.834795:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757424.834798:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834800:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757424.834804:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834809:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305152014 00000100:00000001:1.0:1787757424.834813:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305152705 00000100:00000001:1.0:1787757424.834817:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305152014 00000100:00000001:1.0:1787757424.834819:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305152705 00000100:00000001:1.0:1787757424.834822:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757424.834825:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834827:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757424.834833:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757424/real 1787757424] req@ffff8ffb45c30700 x1874588659646720/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757424.834843:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757424.834845:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757424.834848:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757424.834850:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757424.834852:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.834855:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757424.834857:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757424.834861:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659646720/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757424.834870:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.834873:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757424.834876:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.834878:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757424.834880:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.834882:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757424.834904:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757424.834907:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757424.834912:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757424.834915:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757424.834919:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646720:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757424.834924:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757424.834926:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757424.834928:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.834931:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757424.834932:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757424.834935:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757424.834938:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757424.834942:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757424.834944:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757424.834946:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.834948:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757424.834952:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834958:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757424.834960:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787757424.834965:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757424.834977:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757424.834982:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757424.834985:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757424.834986:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834989:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757424.834992:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.834996:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305152011 00000100:00000001:1.0:1787757424.834999:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305152705 00000100:00000001:1.0:1787757424.835002:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305152011 00000100:00000001:1.0:1787757424.835005:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305152705 00000100:00000001:1.0:1787757424.835008:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757424.835010:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.835012:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757424.835018:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757424/real 1787757424] req@ffff8ffb45c30000 x1874588659646848/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757424.835028:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757424.835030:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757424.835032:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757424.835034:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757424.835036:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835038:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757424.835039:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757424.835044:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659646848/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757424.835053:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835055:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757424.835058:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835060:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757424.835061:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835063:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757424.835082:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757424.835085:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757424.835088:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757424.835091:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757424.835094:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757424.835098:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757424.835101:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757424.835102:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.835105:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757424.835107:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757424.835110:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757424.835113:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646976:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757424.835117:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757424.835120:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757424.835122:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757424.835124:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757424.835128:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.835133:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757424.835135:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757424.835139:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757424.835150:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757424.835154:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757424.835157:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757424.835159:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.835161:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757424.835164:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.835168:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305152011 00000100:00000001:1.0:1787757424.835171:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305152705 00000100:00000001:1.0:1787757424.835174:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305152011 00000100:00000001:1.0:1787757424.835177:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305152705 00000100:00000001:1.0:1787757424.835179:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757424.835214:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.835216:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757424.835222:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757424/real 1787757424] req@ffff8ffb45c30e00 x1874588659646976/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757424.835232:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757424.835233:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757424.835237:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757424.835239:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757424.835240:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835242:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757424.835244:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757424.835249:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659646976/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757424.835257:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835260:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757424.835263:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835265:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757424.835267:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835269:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757424.835288:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757424.835292:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757424.835296:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757424.835298:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757424.835302:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659646976:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757424.835307:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835309:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757424.835310:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757424.835312:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757424.835315:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757424.835317:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757424.835318:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757424.835321:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757424.835323:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757424.835326:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757424.835327:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835330:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757424.835331:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757424.835333:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757424.835337:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757424.835338:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757424.835339:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757424.835341:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757424.835343:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757424.835344:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757424.835346:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835348:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757424.835349:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757424.835350:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757424.835352:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757424.835353:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757424.835354:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757424.835356:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757424.835358:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757424.835361:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757424.835362:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835364:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757424.835369:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757424.835371:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757424.835373:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757424.835375:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757425.409376:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757425.409385:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757425.409388:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757425.409395:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757425.409399:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757425.409404:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757425.409407:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757425.409410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757425.409414:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757425.601387:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757425.601402:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757425.601415:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757425.601418:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757425.601498:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757425.601507:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757425.601514:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757426.433310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757426.433319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757426.433322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757426.433329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757426.433333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757426.433338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757426.433341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757426.433344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757426.433348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757426.625304:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757426.625314:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757426.625324:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757426.625326:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757426.625447:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757426.625453:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757426.625459:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757427.457322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757427.457331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757427.457333:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757427.457339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757427.457344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757427.457349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757427.457352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757427.457355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757427.457359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757427.649345:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757427.649353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757427.649364:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757427.649368:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757427.649514:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757427.649520:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757427.649525:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757428.481366:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757428.481375:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757428.481378:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757428.481385:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757428.481391:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757428.481396:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757428.481400:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757428.481405:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757428.481408:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757428.673303:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757428.673311:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757428.673319:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757428.673322:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757428.673351:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757428.673355:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757428.673358:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757429.505372:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757429.505382:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757429.505385:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757429.505391:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757429.505396:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757429.505401:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757429.505405:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757429.505408:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757429.505411:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757429.697356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757429.697363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757429.697370:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757429.697372:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757429.697483:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757429.697493:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757429.697499:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757429.953320:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757429.953336:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757429.953340:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757429.953346:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757429.953353:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757429.953364:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10484 00000100:00080000:3.0:1787757429.953382:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757429.953387:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757429.953390:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757429.953396:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757429.953400:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757429.953405:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757429.953408:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757429.953411:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757429.953413:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757429.953420:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757429.953424:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757429.953428:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757429.953433:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757429.953438:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953441:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757429.953444:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757429.953464:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757429.953466:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953471:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757429.953475:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757429.953483:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953502:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757429.953517:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659647104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757429.953533:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757429.953539:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953567:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757429.953571:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757429.953576:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757429.953578:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757429.953581:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757429.953584:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757429.953587:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10484 00000100:00080000:3.0:1787757429.953591:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757429.953593:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757429.953595:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757429.953598:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757429.953600:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757429.953627:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757429.953631:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757429.953634:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757429.953636:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757429.953640:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757429.953643:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757429.953646:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757429.953654:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757429.953657:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953660:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757429.953667:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757429.953671:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953674:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757429.953680:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757429.953682:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953687:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757429.953688:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757429.953689:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757429.953697:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.953699:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757429.953704:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.953706:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757429.953713:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757429.953720:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757429.953721:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659647232/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757429.953730:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757429.953738:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757429.953739:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757429.953747:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953748:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953760:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757429.953764:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659647104:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757429.953766:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757429.953779:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757429.953782:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757429.953783:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757429.953790:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757429.953793:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757429.953794:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757429.953800:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757429.953805:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10484 02000000:00000001:1.0:1787757429.953807:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757429.953816:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757429.953819:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787757429.953821:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953822:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757429.953832:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757429.953833:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757429.953837:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757429.953841:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757429.953844:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757429.953846:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757429.953850:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757429.953853:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.953854:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757429.953856:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757429.953860:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 02000000:00000001:3.0:1787757429.953863:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757429.953869:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757429.953870:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953875:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757429.953879:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757429.953884:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953887:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757429.953893:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757429.953895:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953897:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757429.953899:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757429.953902:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953907:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757429.953913:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659647360/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757429.953924:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757429.953926:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.953929:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757429.953933:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757429.953941:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787757429.953942:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757429.953945:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757429.953950:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757429.953954:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757429.953960:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10484 00000100:00080000:3.0:1787757429.953967:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757429.953968:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757429.953971:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757429.953975:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757429.953978:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:3.0:1787757429.953980:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757429.953986:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757429.953987:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757429.953994:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757429.953995:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:3.0:1787757429.954001:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757429.954006:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757429.954010:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.954010:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757429.954024:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757429.954025:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305157825 00000100:00000001:3.0:1787757429.954033:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757429.954039:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757429.954043:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757429.954046:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757429.954047:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.954055:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757429.954058:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757429.954063:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305157825 00000008:00000001:3.0:1787757429.954064:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.954073:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954077:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757429.954082:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757429.954085:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757429.954087:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954093:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.954094:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757429.954101:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757429.954103:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787757429.954105:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757429.954115:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757429.954116:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757429/real 1787757429] req@ffff8ffb45c30a80 x1874588659647104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757429.954126:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659647488/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757429.954157:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757429.954158:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:1.0:1787757429.954162:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757429.954164:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954170:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757429.954171:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954177:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757429.954225:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954232:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757429.954237:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757429.954251:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659647104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757429.954269:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954276:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757429.954283:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954288:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757429.954291:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954296:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757429.954301:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757429.954307:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757429.954316:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757429.954322:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757429.954331:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659647104:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757429.954340:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954344:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757429.954346:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757429.954349:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757429.954357:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757429.954360:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757429.954363:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757429.954371:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757429.954374:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757429.954377:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757429.954379:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954382:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954393:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757429.954396:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954399:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757429.954401:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757429.954403:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757429.954405:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757429.954407:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757429.954411:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757429.954414:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757429.954416:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757429.954421:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647232:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757429.954425:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757429.954428:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757429.954430:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757429.954432:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757429.954437:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954446:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757429.954448:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757429.954454:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757429.954473:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757429.954480:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757429.954483:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757429.954484:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954487:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757429.954491:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954497:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305157134 00000100:00000001:1.0:1787757429.954501:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305157825 00000100:00000001:1.0:1787757429.954504:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305157134 00000100:00000001:1.0:1787757429.954507:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305157825 00000100:00000001:1.0:1787757429.954510:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757429.954512:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954515:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757429.954521:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757429/real 1787757429] req@ffff8ffb45c33b80 x1874588659647232/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757429.954532:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757429.954534:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757429.954537:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757429.954539:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757429.954541:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954543:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757429.954545:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757429.954550:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659647232/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757429.954558:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954561:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757429.954564:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954566:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757429.954567:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954569:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757429.954592:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757429.954595:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757429.954598:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757429.954601:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757429.954636:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647232:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757429.954642:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757429.954644:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757429.954646:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757429.954648:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757429.954651:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757429.954654:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757429.954657:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647360:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757429.954665:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757429.954668:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757429.954670:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757429.954672:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757429.954676:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954685:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757429.954686:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:1.0:1787757429.954692:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757429.954708:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757429.954714:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757429.954717:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757429.954719:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954721:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757429.954724:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954728:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305157131 00000100:00000001:1.0:1787757429.954732:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305157825 00000100:00000001:1.0:1787757429.954734:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305157131 00000100:00000001:1.0:1787757429.954737:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305157825 00000100:00000001:1.0:1787757429.954740:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757429.954742:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954745:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757429.954750:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757429/real 1787757429] req@ffff8ffb45c31c00 x1874588659647360/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757429.954759:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757429.954761:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757429.954764:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757429.954765:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757429.954768:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954770:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757429.954772:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757429.954776:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659647360/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757429.954793:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954795:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757429.954798:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954800:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757429.954802:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954804:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757429.954838:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757429.954841:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757429.954850:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757429.954853:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757429.954857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647360:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757429.954862:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757429.954864:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757429.954866:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757429.954868:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757429.954870:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757429.954873:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757429.954876:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647488:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757429.954880:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757429.954882:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757429.954884:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757429.954887:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757429.954890:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954896:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757429.954898:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757429.954902:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757429.954915:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757429.954921:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757429.954923:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757429.954925:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954928:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757429.954931:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954935:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305157131 00000100:00000001:1.0:1787757429.954938:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305157825 00000100:00000001:1.0:1787757429.954941:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305157131 00000100:00000001:1.0:1787757429.954944:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305157825 00000100:00000001:1.0:1787757429.954947:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757429.954949:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.954951:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757429.954956:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757429/real 1787757429] req@ffff8ffb45c30e00 x1874588659647488/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757429.954967:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757429.954971:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757429.954978:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757429.954981:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757429.954984:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.954988:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757429.954992:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757429.955002:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659647488/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757429.955017:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.955022:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757429.955027:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.955030:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757429.955033:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.955036:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757429.955071:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757429.955077:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757429.955085:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757429.955089:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757429.955097:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647488:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757429.955106:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.955109:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757429.955112:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757429.955115:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757429.955120:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757429.955123:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757429.955126:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757429.955131:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757429.955133:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757429.955137:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757429.955140:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.955143:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757429.955146:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757429.955149:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757429.955154:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757429.955156:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757429.955158:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757429.955162:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757429.955164:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757429.955169:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757429.955171:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.955173:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757429.955175:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757429.955176:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757429.955178:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757429.955180:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757429.955213:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757429.955216:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757429.955218:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757429.955221:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757429.955223:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.955227:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757429.955234:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757429.955237:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757429.955242:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757429.955245:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757430.529342:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757430.529354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757430.529358:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757430.529367:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757430.529374:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757430.529381:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757430.529386:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757430.529391:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757430.529396:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757430.721359:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757430.721370:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757430.721391:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757430.721395:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757430.721474:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757430.721484:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757430.721491:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757431.553320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757431.553329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757431.553332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757431.553338:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757431.553342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757431.553349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757431.553352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757431.553355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757431.553359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757431.745298:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757431.745306:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757431.745314:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757431.745321:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757431.745326:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757431.745331:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757431.745336:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757431.745339:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757431.745341:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757431.745346:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757431.745483:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757431.745489:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757431.745494:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757432.577329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757432.577341:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757432.577345:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757432.577353:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757432.577358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757432.577364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757432.577368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757432.577372:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757432.577376:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757432.769375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757432.769389:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757432.769404:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757432.769407:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757432.769456:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757432.769462:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757432.769469:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757433.601315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757433.601324:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757433.601327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757433.601332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757433.601337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757433.601342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757433.601345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757433.601349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757433.601353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757433.793314:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757433.793325:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757433.793329:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757433.793334:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757433.793341:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757433.793344:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757433.793347:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757433.793355:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757433.793366:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757433.793370:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757433.793475:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757433.793483:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757433.793489:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757434.625317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757434.625326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757434.625328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757434.625335:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757434.625340:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757434.625344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757434.625348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757434.625352:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757434.625355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757434.817347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757434.817355:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757434.817363:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757434.817365:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757434.817565:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757434.817574:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757434.817581:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757435.073306:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757435.073322:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757435.073325:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757435.073331:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757435.073338:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757435.073350:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10489 00000100:00080000:3.0:1787757435.073367:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757435.073373:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757435.073376:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757435.073381:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757435.073385:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757435.073390:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757435.073393:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757435.073396:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757435.073398:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757435.073405:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757435.073409:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757435.073413:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757435.073418:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757435.073422:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073425:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757435.073428:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757435.073446:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757435.073448:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073453:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757435.073458:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757435.073464:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073483:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757435.073497:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659647616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757435.073556:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757435.073564:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073587:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757435.073592:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757435.073597:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757435.073599:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757435.073601:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757435.073605:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757435.073609:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10489 00000100:00080000:3.0:1787757435.073615:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757435.073619:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757435.073621:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757435.073625:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757435.073627:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757435.073631:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757435.073634:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757435.073637:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757435.073638:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757435.073643:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757435.073647:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.073649:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757435.073651:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757435.073661:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757435.073665:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757435.073667:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757435.073675:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.073676:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757435.073681:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757435.073683:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757435.073690:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757435.073693:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:3.0:1787757435.073696:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.073700:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757435.073701:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.073711:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757435.073713:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757435.073717:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073722:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:1.0:1787757435.073723:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659647616:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757435.073725:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757435.073732:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.073737:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757435.073744:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.073747:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757435.073752:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757435.073757:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659647744/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757435.073759:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757435.073771:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073773:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757435.073779:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073784:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757435.073788:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757435.073794:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757435.073797:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757435.073799:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757435.073803:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757435.073804:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757435.073810:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00080000:3.0:1787757435.073816:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10489 00000100:00000001:1.0:1787757435.073820:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757435.073832:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757435.073837:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757435.073841:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757435.073846:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757435.073848:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757435.073854:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757435.073858:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757435.073863:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757435.073866:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757435.073875:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757435.073881:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757435.073885:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000800:00000100:1.0:1787757435.073892:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757435.073895:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757435.073904:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073908:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757435.073914:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00100000:3.0:1787757435.073915:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757435.073923:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00000001:3.0:1787757435.073925:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.073930:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.073931:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.073940:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757435.073946:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757435.073951:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757435.073951:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.073959:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305162945 00000100:00000001:3.0:1787757435.073972:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757435.073975:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757435.073976:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757435.073983:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.073990:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305162945 00000100:00000001:3.0:1787757435.073995:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074000:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757435.074007:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:3.0:1787757435.074010:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659647872/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757435.074013:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074022:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757435.074031:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:1.0:1787757435.074036:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757435/real 1787757435] req@ffff8ffb45c30000 x1874588659647616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757435.074037:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.074047:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757435.074056:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757435.074067:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000400:00000001:1.0:1787757435.074068:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757435.074071:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000400:00000001:1.0:1787757435.074074:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:3.0:1787757435.074076:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757435.074082:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757435.074084:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757435.074089:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787757435.074093:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10489 00000100:00000001:1.0:1787757435.074095:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074103:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757435.074104:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757435.074109:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757435.074111:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757435.074117:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757435.074125:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659647616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757435.074126:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757435.074131:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757435.074137:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757435.074145:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757435.074151:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757435.074154:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757435.074160:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757435.074163:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757435.074165:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757435.074169:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757435.074174:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757435.074175:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.074218:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757435.074221:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757435.074224:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757435.074227:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757435.074232:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757435.074237:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074240:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757435.074241:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757435.074246:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757435.074252:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074253:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074259:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757435.074261:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074262:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074272:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757435.074276:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:3.0:1787757435.074279:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659648000/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757435.074282:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757435.074298:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757435.074304:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757435.074306:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757435.074310:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757435.074317:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659647616:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757435.074318:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074333:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074348:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757435.074351:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757435.074356:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757435.074363:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757435.074367:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757435.074370:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757435.074376:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757435.074378:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757435.074382:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757435.074384:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074387:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074397:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757435.074400:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074403:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757435.074405:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757435.074407:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757435.074409:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757435.074411:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.074414:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757435.074418:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757435.074420:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757435.074425:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647744:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757435.074430:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757435.074433:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757435.074435:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.074437:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757435.074441:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074449:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757435.074451:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757435.074458:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757435.074475:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757435.074481:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757435.074483:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757435.074485:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074487:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757435.074491:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074497:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305162254 00000100:00000001:1.0:1787757435.074501:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305162945 00000100:00000001:1.0:1787757435.074504:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305162254 00000100:00000001:1.0:1787757435.074507:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305162945 00000100:00000001:1.0:1787757435.074510:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757435.074541:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074544:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757435.074550:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757435/real 1787757435] req@ffff8ffb45c30700 x1874588659647744/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757435.074561:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757435.074563:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757435.074567:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757435.074569:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757435.074571:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074573:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757435.074575:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757435.074580:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659647744/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757435.074590:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074593:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757435.074596:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074598:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757435.074599:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074601:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757435.074623:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757435.074626:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757435.074630:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757435.074633:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757435.074637:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647744:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757435.074642:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757435.074644:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757435.074646:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.074649:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757435.074651:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757435.074653:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757435.074657:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647872:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757435.074661:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757435.074663:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757435.074665:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.074667:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757435.074671:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074677:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757435.074679:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787757435.074684:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757435.074698:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757435.074703:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757435.074706:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757435.074708:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074711:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757435.074714:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074718:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305162251 00000100:00000001:1.0:1787757435.074721:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305162945 00000100:00000001:1.0:1787757435.074724:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305162251 00000100:00000001:1.0:1787757435.074727:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305162945 00000100:00000001:1.0:1787757435.074730:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757435.074732:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074734:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757435.074739:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757435/real 1787757435] req@ffff8ffb45c32a00 x1874588659647872/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757435.074748:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757435.074749:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757435.074752:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757435.074754:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757435.074756:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074758:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757435.074760:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757435.074764:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659647872/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757435.074772:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074775:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757435.074777:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074779:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757435.074781:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074783:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757435.074801:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757435.074804:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757435.074807:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757435.074810:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757435.074814:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659647872:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757435.074819:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757435.074821:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757435.074822:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.074825:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757435.074827:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757435.074829:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757435.074832:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648000:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757435.074836:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757435.074837:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757435.074839:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757435.074841:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757435.074845:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074850:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757435.074851:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757435.074856:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757435.074867:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757435.074871:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757435.074874:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757435.074876:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074878:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757435.074881:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074884:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305162251 00000100:00000001:1.0:1787757435.074888:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305162945 00000100:00000001:1.0:1787757435.074890:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305162251 00000100:00000001:1.0:1787757435.074893:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305162945 00000100:00000001:1.0:1787757435.074896:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757435.074898:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.074900:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757435.074904:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757435/real 1787757435] req@ffff8ffb45c33800 x1874588659648000/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757435.074922:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757435.074924:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757435.074926:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757435.074928:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757435.074930:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074931:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757435.074933:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757435.074937:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659648000/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757435.074945:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074948:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757435.074951:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074952:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757435.074954:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074956:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757435.074975:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757435.074978:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757435.074981:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757435.074983:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757435.074987:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648000:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757435.074992:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.074994:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757435.074996:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757435.074998:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757435.075000:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757435.075002:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757435.075004:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757435.075006:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757435.075007:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757435.075010:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757435.075012:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.075014:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757435.075016:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757435.075017:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757435.075021:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757435.075023:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757435.075024:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757435.075026:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757435.075027:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757435.075030:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757435.075031:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.075033:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757435.075034:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757435.075036:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757435.075037:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757435.075039:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757435.075040:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757435.075042:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757435.075043:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757435.075045:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757435.075046:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.075048:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757435.075052:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757435.075054:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757435.075056:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757435.075058:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757435.649294:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757435.649306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757435.649311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757435.649320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757435.649326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757435.649332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757435.649337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757435.649342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757435.649347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757435.841351:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757435.841361:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757435.841372:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757435.841376:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757435.841441:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757435.841451:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757435.841459:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757436.673329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757436.673339:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757436.673342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757436.673349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757436.673353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757436.673358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757436.673362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757436.673366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757436.673369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757436.865346:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757436.865355:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757436.865362:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757436.865364:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757436.865451:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757436.865456:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757436.865460:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757437.697371:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757437.697383:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757437.697386:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757437.697393:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757437.697399:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757437.697404:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757437.697408:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757437.697413:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757437.697416:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757437.889353:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757437.889363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757437.889372:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757437.889374:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757437.889454:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757437.889471:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757437.889476:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757438.721272:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757438.721278:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757438.721280:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757438.721284:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757438.721287:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757438.721290:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757438.721292:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757438.721294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757438.721296:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757438.913280:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757438.913287:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757438.913294:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757438.913295:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757438.913369:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757438.913372:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757438.913375:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757439.745271:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757439.745280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757439.745283:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757439.745290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757439.745294:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757439.745299:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757439.745302:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757439.745305:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757439.745309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757439.937340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757439.937352:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757439.937378:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757439.937381:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757439.937493:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757439.937499:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757439.937504:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757440.193336:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757440.193350:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757440.193353:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757440.193358:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757440.193364:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757440.193372:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10494 00000100:00080000:3.0:1787757440.193387:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757440.193391:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757440.193394:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757440.193399:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757440.193403:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757440.193407:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757440.193410:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757440.193414:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757440.193417:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757440.193451:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757440.193457:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757440.193462:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757440.193469:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757440.193475:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193479:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757440.193483:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757440.193501:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757440.193505:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193511:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757440.193515:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757440.193522:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193542:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757440.193557:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659648128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757440.193572:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757440.193578:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193602:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757440.193606:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757440.193611:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757440.193613:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757440.193616:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757440.193619:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757440.193623:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10494 00000100:00080000:3.0:1787757440.193629:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757440.193632:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757440.193634:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757440.193637:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757440.193640:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757440.193644:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757440.193647:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757440.193651:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757440.193654:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757440.193658:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757440.193663:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757440.193667:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757440.193672:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757440.193675:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757440.193681:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757440.193683:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.193690:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757440.193691:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757440.193698:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:3.0:1787757440.193703:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757440.193704:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757440.193717:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.193719:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757440.193722:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.193727:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757440.193732:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757440.193734:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757440.193738:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193743:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:1.0:1787757440.193749:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659648128:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757440.193750:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757440.193757:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.193765:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757440.193771:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.193774:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757440.193781:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757440.193787:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659648256/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757440.193788:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757440.193799:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193810:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757440.193814:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193820:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757440.193824:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757440.193828:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757440.193830:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757440.193836:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:3.0:1787757440.193838:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757440.193845:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757440.193848:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757440.193852:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757440.193859:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10494 00000100:00080000:3.0:1787757440.193864:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757440.193866:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757440.193868:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757440.193871:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757440.193873:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757440.193876:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757440.193878:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757440.193880:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757440.193884:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757440.193890:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757440.193894:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757440.193897:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757440.193901:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000800:00000100:1.0:1787757440.193910:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757440.193911:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193919:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757440.193922:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757440.193927:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193930:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.193933:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757440.193938:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757440.193942:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757440.193943:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.193950:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193951:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757440.193956:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757440.193958:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757440.193961:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.193970:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.193973:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.193979:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305168065 00000100:00100000:3.0:1787757440.193987:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659648384/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757440.193996:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757440.194003:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757440.194008:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194010:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305168065 00000100:00000001:3.0:1787757440.194013:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757440.194022:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757440.194023:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787757440.194030:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757440.194033:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757440.194036:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757440.194041:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757440.194042:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757440.194052:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757440.194053:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757440.194061:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10494 00000100:00100000:1.0:1787757440.194069:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757440/real 1787757440] req@ffff8ffb45c33480 x1874588659648128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757440.194071:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757440.194079:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757440.194083:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757440.194092:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757440.194097:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757440.194102:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787757440.194107:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:3.0:1787757440.194109:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787757440.194113:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757440.194116:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757440.194121:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757440.194122:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757440.194132:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787757440.194134:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757440.194141:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757440.194143:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757440.194151:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757440.194156:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757440.194158:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757440.194166:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194167:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757440.194169:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757440.194174:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757440.194180:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.194221:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757440.194231:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757440.194234:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757440.194238:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757440.194242:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757440.194249:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757440.194251:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659648128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757440.194267:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194270:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194280:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757440.194288:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757440.194290:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659648512/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757440.194292:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757440.194297:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194307:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757440.194313:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757440.194322:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00080000:1.0:1787757440.194323:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757440.194331:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757440.194339:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757440.194340:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194345:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757440.194355:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659648128:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757440.194362:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194365:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757440.194367:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757440.194371:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757440.194378:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757440.194381:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757440.194384:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757440.194390:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757440.194392:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757440.194395:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757440.194397:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194400:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194409:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757440.194412:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194415:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757440.194417:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757440.194419:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757440.194421:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757440.194456:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.194461:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757440.194464:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757440.194467:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757440.194471:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648256:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757440.194477:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757440.194480:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757440.194482:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.194484:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757440.194489:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194498:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757440.194500:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757440.194506:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757440.194527:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757440.194534:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757440.194537:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757440.194540:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194542:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757440.194546:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194551:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305167374 00000100:00000001:1.0:1787757440.194556:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305168065 00000100:00000001:1.0:1787757440.194559:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305167374 00000100:00000001:1.0:1787757440.194562:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305168065 00000100:00000001:1.0:1787757440.194565:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757440.194568:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194570:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757440.194575:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757440/real 1787757440] req@ffff8ffb45c31180 x1874588659648256/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757440.194586:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757440.194587:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757440.194590:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757440.194593:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757440.194594:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194597:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757440.194599:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757440.194604:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659648256/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757440.194613:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194615:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757440.194618:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194620:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757440.194622:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194624:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757440.194640:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757440.194643:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757440.194648:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757440.194651:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757440.194655:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648256:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757440.194660:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757440.194662:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757440.194664:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.194666:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757440.194668:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757440.194671:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757440.194675:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648384:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757440.194679:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757440.194681:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757440.194683:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.194685:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757440.194688:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194694:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757440.194696:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757440.194700:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757440.194713:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757440.194717:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757440.194721:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757440.194722:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194725:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757440.194728:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194731:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305167371 00000100:00000001:1.0:1787757440.194735:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305168065 00000100:00000001:1.0:1787757440.194737:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305167371 00000100:00000001:1.0:1787757440.194741:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305168065 00000100:00000001:1.0:1787757440.194744:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757440.194746:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194748:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757440.194754:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757440/real 1787757440] req@ffff8ffb45c31500 x1874588659648384/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757440.194764:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757440.194766:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757440.194769:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757440.194771:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757440.194772:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194774:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757440.194776:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757440.194781:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659648384/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757440.194790:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194792:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757440.194795:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194797:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757440.194798:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194800:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757440.194814:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757440.194817:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757440.194820:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757440.194823:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757440.194826:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648384:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757440.194830:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757440.194832:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757440.194834:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.194837:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757440.194839:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757440.194842:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757440.194845:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648512:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757440.194848:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757440.194851:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757440.194853:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757440.194855:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757440.194859:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194864:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757440.194866:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757440.194870:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757440.194881:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757440.194886:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757440.194888:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757440.194890:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194892:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757440.194895:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194899:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305167371 00000100:00000001:1.0:1787757440.194902:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305168065 00000100:00000001:1.0:1787757440.194905:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305167371 00000100:00000001:1.0:1787757440.194908:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305168065 00000100:00000001:1.0:1787757440.194911:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757440.194913:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.194915:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757440.194919:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757440/real 1787757440] req@ffff8ffb45c33800 x1874588659648512/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757440.194928:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757440.194929:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757440.194932:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757440.194933:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757440.194935:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194937:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757440.194938:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757440.194943:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659648512/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757440.194951:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194953:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757440.194957:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194959:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757440.194960:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194962:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757440.194977:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757440.194980:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757440.194984:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757440.194987:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757440.194991:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648512:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757440.194995:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.194997:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757440.194999:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757440.195001:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757440.195003:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757440.195005:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757440.195007:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757440.195009:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757440.195011:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757440.195013:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757440.195015:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.195017:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757440.195019:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757440.195020:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757440.195024:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757440.195025:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757440.195026:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757440.195028:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757440.195029:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757440.195031:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757440.195032:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.195034:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757440.195035:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757440.195037:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757440.195038:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757440.195040:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757440.195041:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757440.195042:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757440.195045:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757440.195046:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757440.195048:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.195050:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757440.195055:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757440.195057:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757440.195060:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757440.195062:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757440.769304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757440.769317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757440.769321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757440.769330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757440.769337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757440.769343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757440.769347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757440.769350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757440.769354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757440.961332:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757440.961343:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757440.961356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757440.961360:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757440.961401:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757440.961407:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757440.961447:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757441.793331:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757441.793341:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757441.793343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757441.793350:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757441.793355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757441.793360:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757441.793363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757441.793367:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757441.793371:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757441.985336:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757441.985345:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757441.985367:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757441.985373:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757441.985377:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757441.985383:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757441.985389:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757441.985418:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757441.985423:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757441.985428:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757441.985504:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757441.985509:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757441.985513:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757442.817456:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757442.817468:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757442.817472:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757442.817481:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757442.817487:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757442.817494:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757442.817499:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757442.817504:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757442.817510:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757443.009361:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757443.009370:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757443.009408:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757443.009410:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757443.009506:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757443.009518:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757443.009527:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757443.841263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757443.841269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757443.841270:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757443.841274:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757443.841277:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757443.841281:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757443.841282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757443.841284:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757443.841286:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757444.033266:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757444.033274:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757444.033288:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757444.033292:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757444.033298:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757444.033300:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757444.033303:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757444.033305:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757444.033310:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757444.033311:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757444.033386:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757444.033390:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757444.033393:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757444.865335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757444.865388:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757444.865391:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757444.865399:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757444.865403:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757444.865411:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757444.865416:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757444.865422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757444.865427:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757445.057433:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757445.057443:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757445.057450:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757445.057452:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757445.057487:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757445.057491:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757445.057494:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757445.313297:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757445.313312:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757445.313316:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757445.313320:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757445.313326:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757445.313364:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10500 00000100:00080000:3.0:1787757445.313378:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757445.313383:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757445.313385:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757445.313392:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757445.313395:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757445.313400:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757445.313403:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757445.313407:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757445.313410:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757445.313417:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757445.313420:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.313423:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757445.313428:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757445.313431:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313434:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757445.313438:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.313450:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757445.313453:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313457:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757445.313460:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757445.313466:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313479:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757445.313492:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659648640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757445.313506:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757445.313512:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313532:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757445.313537:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757445.313544:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757445.313546:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757445.313549:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757445.313553:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757445.313556:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10500 00000100:00080000:3.0:1787757445.313561:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757445.313564:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757445.313566:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757445.313569:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757445.313571:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757445.313575:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757445.313578:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757445.313582:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757445.313584:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757445.313589:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757445.313592:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757445.313593:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.313601:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757445.313606:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757445.313610:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757445.313615:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757445.313617:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313624:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757445.313625:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:3.0:1787757445.313631:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757445.313634:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.313645:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757445.313650:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787757445.313654:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.313663:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313665:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.313682:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:1.0:1787757445.313687:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659648640:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757445.313688:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313698:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757445.313704:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757445.313708:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757445.313712:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.313719:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757445.313728:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313729:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.313737:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787757445.313748:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659648768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757445.313752:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.313785:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757445.313792:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757445.313797:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757445.313805:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.313807:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757445.313811:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757445.313818:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757445.313823:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757445.313826:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757445.313829:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757445.313833:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757445.313837:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10500 00000100:00080000:3.0:1787757445.313843:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757445.313846:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757445.313849:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757445.313853:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757445.313855:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757445.313858:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757445.313861:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757445.313864:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757445.313866:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:1.0:1787757445.313877:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757445.313878:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757445.313889:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.313892:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757445.313897:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757445.313900:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.313903:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757445.313906:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00100000:3.0:1787757445.313908:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757445.313916:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.313921:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757445.313921:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.313932:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.313934:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757445.313939:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.313943:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757445.313945:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757445.313953:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757445.313957:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.313958:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.313972:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305173185 00000100:00000001:3.0:1787757445.313993:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.313996:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757445.314013:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305173185 00000100:00100000:3.0:1787757445.314014:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659648896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757445.314028:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757445.314039:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757445.314043:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757445.314053:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.314055:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.314067:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314068:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:3.0:1787757445.314072:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757445.314079:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757445.314085:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00100000:1.0:1787757445.314089:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757445/real 1787757445] req@ffff8ffb45c32a00 x1874588659648640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757445.314090:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757445.314100:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757445.314110:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10500 00000100:00080000:3.0:1787757445.314121:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757445.314128:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757445.314135:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:1.0:1787757445.314140:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:3.0:1787757445.314140:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757445.314147:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000400:00000001:1.0:1787757445.314150:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000020:00000001:3.0:1787757445.314152:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757445.314162:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757445.314164:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757445.314169:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757445.314174:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757445.314177:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757445.314234:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314246:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757445.314253:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757445.314255:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757445.314264:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.314270:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:1.0:1787757445.314279:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659648640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757445.314279:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757445.314288:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.314295:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757445.314300:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757445.314310:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.314317:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757445.314330:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757445.314331:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757445.314360:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.314363:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757445.314368:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757445.314374:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314379:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757445.314385:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314391:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314401:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:3.0:1787757445.314401:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659649024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757445.314408:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314420:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757445.314429:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757445.314431:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757445.314434:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757445.314442:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757445.314448:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757445.314462:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757445.314470:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757445.314480:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659648640:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757445.314491:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314496:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757445.314499:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757445.314503:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757445.314511:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757445.314515:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757445.314519:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757445.314525:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757445.314529:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757445.314534:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757445.314537:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314542:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314553:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757445.314557:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314561:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757445.314565:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757445.314568:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757445.314571:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757445.314574:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.314579:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757445.314583:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757445.314587:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757445.314594:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648768:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757445.314602:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757445.314606:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757445.314609:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.314612:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757445.314618:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314632:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757445.314634:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757445.314642:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757445.314669:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757445.314679:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757445.314684:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757445.314687:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314690:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757445.314697:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314704:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305172494 00000100:00000001:1.0:1787757445.314711:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305173185 00000100:00000001:1.0:1787757445.314716:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305172494 00000100:00000001:1.0:1787757445.314720:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305173185 00000100:00000001:1.0:1787757445.314725:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757445.314729:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314734:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757445.314744:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757445/real 1787757445] req@ffff8ffb45c30700 x1874588659648768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757445.314759:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757445.314761:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757445.314765:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757445.314768:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757445.314770:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314773:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757445.314775:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757445.314780:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659648768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757445.314790:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314794:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757445.314797:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314799:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757445.314801:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314803:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757445.314819:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757445.314822:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757445.314826:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757445.314828:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757445.314833:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648768:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757445.314838:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757445.314840:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757445.314842:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.314845:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757445.314847:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757445.314850:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757445.314854:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757445.314862:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757445.314865:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757445.314867:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.314869:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757445.314874:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314881:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757445.314883:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787757445.314888:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757445.314905:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757445.314911:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757445.314914:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757445.314916:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314918:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757445.314922:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314926:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305172491 00000100:00000001:1.0:1787757445.314930:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305173185 00000100:00000001:1.0:1787757445.314933:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305172491 00000100:00000001:1.0:1787757445.314936:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305173185 00000100:00000001:1.0:1787757445.314939:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757445.314941:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.314943:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757445.314948:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757445/real 1787757445] req@ffff8ffb45c30000 x1874588659648896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757445.314957:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757445.314958:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757445.314961:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757445.314963:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757445.314965:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314968:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757445.314969:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757445.314974:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659648896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757445.314992:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.314995:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757445.314998:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315000:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757445.315001:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315003:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757445.315015:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757445.315018:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757445.315022:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757445.315024:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757445.315028:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659648896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757445.315033:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757445.315034:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757445.315036:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.315039:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757445.315042:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757445.315044:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757445.315047:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757445.315051:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757445.315053:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757445.315054:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757445.315056:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757445.315060:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.315064:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757445.315066:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757445.315070:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757445.315082:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757445.315087:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757445.315089:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757445.315091:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.315094:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757445.315097:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.315101:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305172491 00000100:00000001:1.0:1787757445.315104:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305173185 00000100:00000001:1.0:1787757445.315107:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305172491 00000100:00000001:1.0:1787757445.315110:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305173185 00000100:00000001:1.0:1787757445.315113:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757445.315115:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.315117:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757445.315121:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757445/real 1787757445] req@ffff8ffb45c30e00 x1874588659649024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757445.315130:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757445.315132:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757445.315134:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757445.315136:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757445.315137:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315139:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757445.315141:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757445.315145:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659649024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757445.315154:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315156:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757445.315160:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315162:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757445.315163:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315165:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757445.315177:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757445.315180:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757445.315214:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757445.315217:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757445.315223:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757445.315227:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315230:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757445.315232:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757445.315234:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757445.315238:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757445.315240:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757445.315242:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757445.315245:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757445.315247:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757445.315249:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757445.315251:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315253:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757445.315255:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757445.315257:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757445.315260:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757445.315261:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757445.315262:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757445.315264:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757445.315265:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757445.315267:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757445.315268:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315270:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757445.315272:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757445.315273:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757445.315274:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757445.315276:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757445.315277:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757445.315279:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757445.315280:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757445.315282:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757445.315283:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315286:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757445.315290:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757445.315292:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757445.315294:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757445.315297:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757445.889304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757445.889313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757445.889315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757445.889368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757445.889374:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757445.889380:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757445.889383:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757445.889387:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757445.889390:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757446.081346:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757446.081358:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757446.081372:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757446.081376:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757446.081434:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757446.081443:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757446.081448:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757446.913487:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757446.913500:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757446.913504:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757446.913513:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757446.913518:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757446.913525:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757446.913529:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757446.913533:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757446.913537:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757447.105418:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757447.105429:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757447.105438:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757447.105440:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757447.105527:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757447.105535:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757447.105540:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757447.937402:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757447.937414:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757447.937417:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757447.937425:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757447.937430:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757447.937436:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757447.937440:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757447.937445:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757447.937448:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757448.129376:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757448.129387:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757448.129396:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757448.129398:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757448.129494:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757448.129500:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757448.129505:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757448.961257:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757448.961265:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757448.961267:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757448.961286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757448.961290:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757448.961294:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757448.961313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757448.961316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757448.961320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757449.153317:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757449.153324:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757449.153330:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757449.153331:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757449.153350:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757449.153353:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757449.153356:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757449.985370:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757449.985378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757449.985381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757449.985388:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757449.985392:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757449.985397:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757449.985400:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757449.985403:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757449.985407:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757450.177336:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757450.177345:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757450.177354:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757450.177357:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757450.177436:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757450.177442:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757450.177447:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757450.433371:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757450.433384:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757450.433387:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757450.433392:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757450.433397:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757450.433407:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10505 00000100:00080000:3.0:1787757450.433421:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757450.433426:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757450.433428:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757450.433433:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757450.433436:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757450.433440:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757450.433443:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757450.433446:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757450.433449:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757450.433454:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757450.433458:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.433462:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757450.433480:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757450.433484:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433487:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757450.433490:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.433506:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757450.433508:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433512:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757450.433515:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757450.433520:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433536:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757450.433548:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659649152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757450.433563:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757450.433568:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433592:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757450.433597:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757450.433601:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757450.433603:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757450.433606:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757450.433609:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757450.433612:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10505 00000100:00080000:3.0:1787757450.433624:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757450.433626:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757450.433628:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757450.433631:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757450.433633:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757450.433636:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757450.433638:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757450.433642:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757450.433644:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757450.433648:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757450.433651:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.433653:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757450.433660:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757450.433664:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433666:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757450.433674:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757450.433680:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433683:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.433688:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757450.433690:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433694:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757450.433696:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757450.433697:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757450.433704:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433715:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.433716:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757450.433728:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:3.0:1787757450.433731:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659649280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757450.433739:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757450.433750:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433751:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757450.433759:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433771:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.433774:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757450.433776:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757450.433785:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787757450.433788:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757450.433789:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757450.433795:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757450.433806:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757450.433808:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757450.433812:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10505 00000100:00080000:3.0:1787757450.433822:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757450.433825:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757450.433828:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757450.433836:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:1.0:1787757450.433838:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659649152:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757450.433838:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757450.433843:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757450.433850:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757450.433864:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757450.433867:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757450.433869:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757450.433877:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757450.433882:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.433887:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757450.433888:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757450.433901:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757450.433903:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.433907:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433916:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757450.433917:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00100000:3.0:1787757450.433920:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757450.433927:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433929:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.433936:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757450.433938:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.433941:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433942:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757450.433952:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757450.433961:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.433974:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.433979:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757450.433983:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00100000:3.0:1787757450.433991:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659649408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757450.433994:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757450.434015:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757450.434021:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757450.434028:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757450.434034:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757450.434042:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757450.434046:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757450.434050:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757450.434057:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757450.434062:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10505 00000800:00000100:1.0:1787757450.434074:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757450.434076:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757450.434093:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757450.434095:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757450.434098:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757450.434100:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757450.434104:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757450.434107:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757450.434110:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757450.434112:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757450.434117:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757450.434118:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757450.434127:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.434131:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757450.434135:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757450.434136:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757450.434145:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.434148:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434155:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757450.434158:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757450.434166:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757450.434173:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757450.434174:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434230:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305178306 00000100:00000001:3.0:1787757450.434270:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434275:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757450.434286:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757450.434289:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305178306 02000000:00000001:3.0:1787757450.434294:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434300:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757450.434305:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757450.434309:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757450.434313:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757450.434316:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757450.434323:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434324:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757450.434340:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757450/real 1787757450] req@ffff8ffb45c31c00 x1874588659649152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757450.434343:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757450.434366:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659649536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757450.434373:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757450.434379:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757450.434392:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757450.434393:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757450.434406:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757450.434407:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434423:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757450.434426:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434442:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757450.434446:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757450.434458:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659649152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757450.434471:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434475:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757450.434481:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434484:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757450.434488:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434492:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757450.434496:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757450.434501:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757450.434507:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757450.434511:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757450.434516:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659649152:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757450.434522:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434527:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757450.434530:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757450.434535:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757450.434543:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757450.434548:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757450.434552:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757450.434559:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757450.434564:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757450.434568:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757450.434570:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434573:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434585:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757450.434589:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434594:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757450.434597:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757450.434599:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757450.434601:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757450.434604:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.434608:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757450.434613:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757450.434617:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757450.434624:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649280:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757450.434630:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757450.434634:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757450.434637:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.434640:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757450.434644:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434654:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757450.434656:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757450.434664:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757450.434690:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757450.434703:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757450.434714:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757450.434721:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434726:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757450.434733:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434740:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305177614 00000100:00000001:1.0:1787757450.434745:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305178306 00000100:00000001:1.0:1787757450.434750:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305177614 00000100:00000001:1.0:1787757450.434754:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305178306 00000100:00000001:1.0:1787757450.434758:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757450.434762:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434766:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757450.434775:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757450/real 1787757450] req@ffff8ffb45c33b80 x1874588659649280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757450.434787:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757450.434789:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757450.434792:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757450.434794:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757450.434795:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434798:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757450.434801:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757450.434807:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659649280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757450.434817:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434820:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757450.434825:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434828:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757450.434830:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.434833:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757450.434878:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757450.434884:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757450.434892:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757450.434896:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757450.434903:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649280:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757450.434911:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757450.434914:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757450.434916:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.434919:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757450.434922:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757450.434924:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757450.434928:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757450.434932:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757450.434934:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757450.434936:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.434938:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757450.434943:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434949:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757450.434951:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757450.434956:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757450.434970:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757450.434975:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757450.434978:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757450.434980:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434983:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757450.434986:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.434990:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305177611 00000100:00000001:1.0:1787757450.434993:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305178306 00000100:00000001:1.0:1787757450.434996:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305177611 00000100:00000001:1.0:1787757450.435000:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305178306 00000100:00000001:1.0:1787757450.435004:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757450.435007:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.435009:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757450.435014:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757450/real 1787757450] req@ffff8ffb45c30a80 x1874588659649408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757450.435025:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757450.435026:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757450.435030:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757450.435032:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757450.435038:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435041:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757450.435045:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757450.435055:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659649408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757450.435068:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435075:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757450.435079:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435083:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757450.435086:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435089:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757450.435113:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757450.435119:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757450.435129:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757450.435134:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757450.435143:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757450.435151:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757450.435154:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757450.435157:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.435162:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757450.435167:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757450.435170:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757450.435175:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757450.435180:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757450.435233:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757450.435235:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757450.435238:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757450.435270:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.435282:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757450.435283:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757450.435290:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757450.435313:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757450.435324:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757450.435330:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757450.435335:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.435339:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757450.435343:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.435349:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305177612 00000100:00000001:1.0:1787757450.435354:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305178306 00000100:00000001:1.0:1787757450.435358:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305177612 00000100:00000001:1.0:1787757450.435364:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305178306 00000100:00000001:1.0:1787757450.435368:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757450.435372:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.435375:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757450.435387:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757450/real 1787757450] req@ffff8ffb45c30e00 x1874588659649536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757450.435406:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757450.435408:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757450.435414:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757450.435423:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757450.435428:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435431:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757450.435434:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757450.435439:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659649536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757450.435454:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435459:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757450.435463:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435466:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757450.435469:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435472:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757450.435581:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757450.435586:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757450.435592:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757450.435595:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757450.435600:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757450.435605:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435607:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757450.435609:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757450.435612:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757450.435616:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757450.435618:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757450.435620:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757450.435623:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757450.435625:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757450.435629:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757450.435630:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435633:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757450.435635:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757450.435637:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757450.435640:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757450.435642:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757450.435644:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757450.435646:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757450.435647:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757450.435649:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757450.435651:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435653:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757450.435654:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757450.435655:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757450.435657:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757450.435658:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757450.435659:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757450.435661:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757450.435662:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757450.435664:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757450.435666:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435668:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757450.435673:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757450.435674:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757450.435677:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757450.435679:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757451.009285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757451.009292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757451.009294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757451.009298:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757451.009301:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757451.009304:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757451.009306:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757451.009309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757451.009311:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757451.201345:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757451.201356:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757451.201367:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757451.201373:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757451.201379:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757451.201386:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757451.201392:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757451.201397:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757451.201401:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757451.201405:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757451.201486:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757451.201492:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757451.201497:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757452.033311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757452.033320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757452.033323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757452.033329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757452.033334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757452.033339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757452.033342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757452.033346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757452.033349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757452.225338:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757452.225349:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757452.225359:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757452.225362:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757452.225476:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757452.225485:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757452.225493:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757453.057405:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757453.057417:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757453.057421:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757453.057428:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757453.057433:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757453.057440:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757453.057444:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757453.057448:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757453.057453:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757453.249446:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757453.249460:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757453.249465:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757453.249469:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757453.249477:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757453.249480:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757453.249483:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757453.249486:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757453.249492:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757453.249494:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757453.249532:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757453.249536:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757453.249540:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757454.081356:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757454.081367:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757454.081370:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757454.081378:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757454.081383:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757454.081389:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757454.081392:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757454.081398:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757454.081403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757454.273281:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757454.273288:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757454.273293:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757454.273294:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757454.273364:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757454.273368:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757454.273371:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757455.105360:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757455.105370:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757455.105373:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757455.105380:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757455.105385:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757455.105391:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757455.105397:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757455.105402:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757455.105407:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757455.297350:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757455.297361:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757455.297371:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757455.297374:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757455.297475:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757455.297482:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757455.297487:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757455.553285:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757455.553299:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757455.553302:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757455.553307:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757455.553313:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757455.553322:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10510 00000100:00080000:3.0:1787757455.553335:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757455.553339:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757455.553342:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757455.553347:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757455.553351:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757455.553355:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757455.553358:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757455.553361:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757455.553363:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757455.553368:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757455.553372:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757455.553376:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757455.553381:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757455.553384:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553387:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757455.553390:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757455.553403:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757455.553406:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553409:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757455.553413:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757455.553418:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553433:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757455.553446:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659649664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757455.553460:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757455.553465:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553487:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757455.553491:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757455.553496:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757455.553498:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757455.553500:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757455.553504:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757455.553507:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10510 00000100:00080000:3.0:1787757455.553511:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757455.553513:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757455.553515:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757455.553518:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757455.553520:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757455.553523:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757455.553527:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757455.553530:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757455.553532:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757455.553536:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757455.553541:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757455.553543:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757455.553549:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757455.553553:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553555:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757455.553559:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787757455.553562:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757455.553567:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:3.0:1787757455.553570:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.553574:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757455.553578:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.553581:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757455.553588:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757455.553593:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757455.553596:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757455.553601:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.553604:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757455.553613:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757455.553614:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757455.553623:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757455.553629:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659649664:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757455.553633:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.553642:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757455.553651:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.553654:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757455.553663:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757455.553673:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659649792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757455.553674:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757455.553688:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553703:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757455.553707:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553713:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757455.553717:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757455.553721:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757455.553722:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757455.553730:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:3.0:1787757455.553731:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757455.553735:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757455.553745:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757455.553748:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757455.553751:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10510 00000100:00080000:3.0:1787757455.553762:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757455.553765:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757455.553768:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757455.553771:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757455.553774:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757455.553777:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757455.553782:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757455.553786:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757455.553790:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757455.553799:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757455.553806:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757455.553814:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757455.553816:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757455.553827:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757455.553833:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757455.553835:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.553844:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757455.553845:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757455.553850:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.553856:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:3.0:1787757455.553856:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757455.553863:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.553869:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305183425 00000100:00000001:1.0:1787757455.553881:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00000001:3.0:1787757455.553885:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553890:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.553893:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305183425 02000000:00000001:3.0:1787757455.553900:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757455.553904:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757455.553906:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.553915:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757455.553917:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757455.553919:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757455.553928:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757455.553929:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.553939:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757455.553946:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757455.553950:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757455/real 1787757455] req@ffff8ffb45c30000 x1874588659649664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757455.553962:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659649920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757455.553982:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757455.553988:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757455.553994:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757455.554002:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757455.554002:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757455.554011:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757455.554014:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554016:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757455.554021:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757455.554028:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757455.554034:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659649664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757455.554039:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757455.554059:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757455.554061:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757455.554065:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757455.554069:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757455.554074:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757455.554076:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554084:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757455.554085:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757455.554087:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554094:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00080000:3.0:1787757455.554099:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10510 00000100:00000001:1.0:1787757455.554100:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757455.554111:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757455.554113:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757455.554121:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757455.554124:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757455.554129:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757455.554131:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757455.554139:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:1.0:1787757455.554142:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659649664:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757455.554144:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757455.554148:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757455.554179:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757455.554228:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757455.554231:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757455.554234:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757455.554237:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757455.554240:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757455.554248:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00080000:3.0:1787757455.554252:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757455.554257:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757455.554261:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757455.554263:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:3.0:1787757455.554266:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000020:00000001:1.0:1787757455.554270:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757455.554275:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757455.554281:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757455.554283:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.554288:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757455.554290:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757455.554296:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000008:00100000:3.0:1787757455.554299:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757455.554303:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787757455.554306:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554309:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757455.554313:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554322:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787757455.554324:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757455.554330:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757455.554331:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.554343:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757455.554344:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757455.554347:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757455.554351:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757455.554353:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554356:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757455.554362:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757455.554364:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.554366:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.554369:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757455.554376:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00100000:3.0:1787757455.554382:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659650048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757455.554383:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757455.554400:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649792:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757455.554413:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757455.554415:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757455.554421:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.554423:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757455.554428:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757455.554431:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.554432:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757455.554442:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554455:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757455.554459:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757455.554466:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757455.554487:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757455.554493:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757455.554497:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757455.554499:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554501:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757455.554505:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554510:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305182734 00000100:00000001:1.0:1787757455.554514:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305183425 00000100:00000001:1.0:1787757455.554517:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305182734 00000100:00000001:1.0:1787757455.554520:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305183425 00000100:00000001:1.0:1787757455.554523:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757455.554525:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554527:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757455.554533:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757455/real 1787757455] req@ffff8ffb45c30700 x1874588659649792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757455.554544:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757455.554545:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757455.554548:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757455.554550:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757455.554552:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554555:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757455.554557:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757455.554561:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659649792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757455.554570:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554573:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757455.554576:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554578:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757455.554579:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554582:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757455.554597:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757455.554601:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757455.554605:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757455.554608:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757455.554612:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649792:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757455.554617:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757455.554619:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757455.554621:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.554624:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757455.554625:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757455.554628:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757455.554631:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757455.554636:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757455.554639:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757455.554642:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.554645:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757455.554650:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554661:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757455.554664:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787757455.554671:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757455.554693:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757455.554702:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757455.554706:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757455.554709:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554712:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757455.554717:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554723:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305182731 00000100:00000001:1.0:1787757455.554729:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305183425 00000100:00000001:1.0:1787757455.554733:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305182731 00000100:00000001:1.0:1787757455.554737:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305183425 00000100:00000001:1.0:1787757455.554742:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757455.554747:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554750:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757455.554759:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757455/real 1787757455] req@ffff8ffb45c32a00 x1874588659649920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757455.554770:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757455.554772:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757455.554776:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757455.554778:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757455.554780:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554782:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757455.554784:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757455.554789:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659649920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757455.554798:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554801:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757455.554804:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554806:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757455.554807:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554810:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757455.554825:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757455.554828:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757455.554832:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757455.554834:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757455.554838:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659649920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757455.554843:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554845:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757455.554847:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757455.554850:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757455.554853:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757455.554855:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757455.554857:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757455.554860:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757455.554861:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757455.554863:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757455.554866:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554868:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757455.554869:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757455.554871:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757455.554874:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757455.554876:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757455.554877:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757455.554879:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757455.554880:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757455.554882:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757455.554884:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554886:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.554891:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757455.554892:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554895:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757455.554897:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757455.554899:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757455.554900:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757455.554902:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.554905:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757455.554907:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757455.554909:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757455.554914:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757455.554918:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757455.554920:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757455.554922:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757455.554924:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757455.554928:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554935:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757455.554937:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757455.554942:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757455.554957:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757455.554963:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757455.554966:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757455.554968:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554970:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757455.554974:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554978:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305182731 00000100:00000001:1.0:1787757455.554982:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305183425 00000100:00000001:1.0:1787757455.554985:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305182731 00000100:00000001:1.0:1787757455.554987:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305183425 00000100:00000001:1.0:1787757455.554990:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757455.554993:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.554995:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757455.555000:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757455/real 1787757455] req@ffff8ffb45c33800 x1874588659650048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757455.555009:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757455.555010:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757455.555013:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757455.555015:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757455.555017:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.555019:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757455.555020:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757455.555025:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659650048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757455.555033:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.555036:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757455.555039:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.555041:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757455.555042:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.555044:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757455.555059:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757455.555061:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757455.555065:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757455.555068:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757455.555072:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757455.555077:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.555079:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757455.555081:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757455.555083:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757455.555085:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757455.555087:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757455.555089:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757455.555091:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757455.555092:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757455.555095:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757455.555096:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.555099:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757455.555102:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757455.555104:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757455.555106:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757455.555109:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757456.129452:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757456.129464:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757456.129468:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757456.129475:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757456.129481:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757456.129487:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757456.129491:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757456.129495:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757456.129499:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757456.321318:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757456.321328:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757456.321337:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757456.321340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757456.321393:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757456.321402:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757456.321409:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757457.153289:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757457.153296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757457.153298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757457.153304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757457.153308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757457.153311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757457.153315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757457.153318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757457.153321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757457.345279:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757457.345285:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757457.345290:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757457.345291:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757457.345310:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757457.345312:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757457.345314:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757458.177246:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757458.177251:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757458.177252:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757458.177256:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757458.177258:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757458.177261:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757458.177263:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757458.177265:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757458.177267:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757458.369296:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757458.369303:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757458.369308:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757458.369310:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757458.369367:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757458.369372:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757458.369375:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757459.201306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757459.201316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757459.201319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757459.201325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757459.201330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757459.201335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757459.201338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757459.201342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757459.201345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757459.393314:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757459.393323:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757459.393333:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757459.393336:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757459.393492:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757459.393498:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757459.393503:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757460.225324:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757460.225333:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757460.225337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757460.225343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757460.225348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757460.225353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757460.225356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757460.225360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757460.225364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757460.417345:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757460.417354:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757460.417362:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757460.417364:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757460.417473:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757460.417482:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757460.417489:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757460.673290:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757460.673303:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757460.673306:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757460.673311:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757460.673317:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757460.673326:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10515 00000100:00080000:3.0:1787757460.673341:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757460.673345:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757460.673348:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757460.673353:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757460.673357:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757460.673360:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757460.673364:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757460.673368:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757460.673371:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757460.673380:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757460.673385:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.673390:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757460.673396:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757460.673401:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673405:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757460.673410:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.673426:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757460.673429:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673436:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757460.673440:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757460.673447:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673465:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757460.673482:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659650176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757460.673503:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757460.673511:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673538:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757460.673545:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757460.673553:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757460.673556:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757460.673560:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757460.673566:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757460.673572:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10515 00000100:00080000:3.0:1787757460.673582:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757460.673587:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757460.673591:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757460.673595:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757460.673597:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757460.673601:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757460.673605:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:3.0:1787757460.673606:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757460.673613:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757460.673615:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757460.673617:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757460.673622:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:3.0:1787757460.673628:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757460.673630:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757460.673637:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673639:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757460.673646:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757460.673646:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757460.673655:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757460.673655:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757460.673668:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757460.673669:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673678:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:1.0:1787757460.673687:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659650176:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1787757460.673689:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757460.673702:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.673704:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757460.673710:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757460.673714:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757460.673719:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.673720:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757460.673723:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757460.673726:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673731:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757460.673734:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757460.673735:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.673738:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673753:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.673763:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:3.0:1787757460.673764:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659650304/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757460.673770:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757460.673783:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757460.673790:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757460.673795:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673800:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757460.673805:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757460.673809:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757460.673812:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757460.673814:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757460.673819:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757460.673823:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10515 00000100:00080000:3.0:1787757460.673830:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757460.673832:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757460.673834:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757460.673837:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757460.673838:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757460.673841:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757460.673844:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757460.673847:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757460.673849:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757460.673853:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757460.673856:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.673859:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757460.673863:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.673864:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757460.673873:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673878:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757460.673881:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305188545 00000008:00100000:3.0:1787757460.673882:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757460.673894:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673897:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.673905:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757460.673908:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757460.673909:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673915:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757460.673919:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757460.673922:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.673926:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305188545 00000100:00000001:3.0:1787757460.673929:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757460.673945:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659650432/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757460.673946:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757460.673960:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757460.673963:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757460.673968:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673970:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.673977:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.673980:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.673987:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757460.673989:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757460.673998:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757460.674000:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757460.674004:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:3.0:1787757460.674005:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757460.674015:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757460.674016:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00080000:3.0:1787757460.674020:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10515 00000100:00000001:1.0:1787757460.674030:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787757460.674031:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757460.674039:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757460.674041:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757460.674043:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757460.674051:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757460.674059:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757460.674087:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757460.674092:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:1.0:1787757460.674095:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:3.0:1787757460.674099:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757460.674105:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787757460.674106:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757460.674116:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:1.0:1787757460.674120:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.674121:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757460.674127:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757460.674139:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1787757460.674141:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650304:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757460.674144:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.674152:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757460.674160:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757460.674161:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757460.674169:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757460.674169:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757460.674178:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.674179:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757460.674218:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757460.674221:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.674223:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757460.674231:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757460.674233:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.674237:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757460.674239:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757460.674242:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.674252:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757460.674261:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659650560/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757460.674265:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757460.674279:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.674280:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757460.674288:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757460.674295:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674303:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757460.674310:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757460.674320:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757460.674351:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674359:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305187854 00000100:00000001:1.0:1787757460.674364:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305188545 00000100:00000001:1.0:1787757460.674369:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305187854 00000100:00000001:1.0:1787757460.674373:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305188545 00000100:00000001:1.0:1787757460.674377:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757460.674382:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674386:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757460.674389:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757460.674392:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757460.674396:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757460.674401:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757460.674406:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757460.674416:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757460.674423:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757460.674427:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757460.674430:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757460.674433:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757460.674438:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674448:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757460.674451:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:1.0:1787757460.674457:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757460.674477:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674483:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305187851 00000100:00000001:1.0:1787757460.674488:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305188545 00000100:00000001:1.0:1787757460.674492:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305187851 00000100:00000001:1.0:1787757460.674497:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305188545 00000100:00000001:1.0:1787757460.674501:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757460.674505:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674509:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674514:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757460.674521:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757460.674524:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757460.674528:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757460.674531:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757460.674533:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757460.674536:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757460.674538:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757460.674543:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757460.674546:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757460.674550:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757460.674557:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757460.674564:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757460.674567:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757460.674571:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757460.674576:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757460.674580:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674589:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757460.674592:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757460.674598:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757460.674616:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674622:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305187851 00000100:00000001:1.0:1787757460.674626:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305188545 00000100:00000001:1.0:1787757460.674630:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305187851 00000100:00000001:1.0:1787757460.674635:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305188545 00000100:00000001:1.0:1787757460.674639:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757460.674642:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674647:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674651:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757460.674656:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757460.674659:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757460.674662:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757460.674665:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757460.674668:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757460.674672:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757461.249276:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757461.249285:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757461.249288:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757461.249294:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757461.249299:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757461.249304:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757461.249307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757461.249310:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757461.249314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757461.441356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757461.441367:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757461.441377:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757461.441383:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757461.441387:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757461.441394:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757461.441400:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757461.441404:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757461.441408:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757461.441413:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757461.441449:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757461.441455:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757461.441460:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757462.273309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757462.273319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757462.273340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757462.273349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757462.273355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757462.273363:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757462.273368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757462.273373:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757462.273379:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757462.465294:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757462.465306:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757462.465319:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757462.465323:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757462.465426:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757462.465450:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757462.465458:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757463.297310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757463.297320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757463.297322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757463.297329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757463.297334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757463.297339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757463.297342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757463.297345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757463.297348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757463.489336:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757463.489346:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757463.489349:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757463.489353:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757463.489359:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757463.489362:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757463.489365:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757463.489368:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757463.489372:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757463.489374:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757463.489465:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757463.489471:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757463.489476:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787757463.745543:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:0.0:1787757463.745608:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787757463.745645:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:0.0:1787757463.745664:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757463.745683:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757463.745689:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757463.745693:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:0.0:1787757463.745709:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:0.0:1787757463.745719:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757463.745726:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757463.745729:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757463.745734:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:0.0:1787757463.745745:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:0.0:1787757463.745755:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757463.745761:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757463.745764:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757463.745769:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757463.745770:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:0.0:1787757463.745780:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:0.0:1787757463.745789:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757463.745797:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757463.745799:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757463.745808:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757463.745809:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:0.0:1787757463.745813:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:1.0:1787757463.745835:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757460/real 1787757463] req@ffff8ffb45c33800 x1874588659650560/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757463.745864:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757463.745868:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757463.745879:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757463.745884:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757463.745888:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.745893:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757463.745897:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757463.745907:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659650560/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757463.745925:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.745931:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757463.745938:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.745942:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757463.745944:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.745979:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757463.746036:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757463.746044:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757463.746052:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757463.746057:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757463.746065:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757463.746076:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757463.746084:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757460/real 1787757463] req@ffff8ffb45c31180 x1874588659650304/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757463.746104:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757463.746107:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757463.746113:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757463.746117:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757463.746120:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746123:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757463.746126:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757463.746135:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659650304/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757463.746150:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746154:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757463.746159:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746162:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757463.746165:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746169:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757463.746233:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757463.746240:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757463.746249:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757463.746254:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757463.746261:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650304:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757463.746269:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757463.746277:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757460/real 1787757463] req@ffff8ffb45c33480 x1874588659650432/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757463.746293:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757463.746296:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757463.746302:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757463.746306:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757463.746309:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746313:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757463.746315:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757463.746324:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659650432/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757463.746338:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746343:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757463.746348:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746351:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757463.746353:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746356:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757463.746378:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757463.746382:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757463.746389:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757463.746394:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757463.746400:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757463.746408:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757463.746416:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757460/real 1787757463] req@ffff8ffb45c31500 x1874588659650176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757463.746433:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757463.746436:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757463.746442:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757463.746449:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757463.746453:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746458:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757463.746461:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757463.746471:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659650176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757463.746486:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746490:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757463.746495:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746498:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757463.746500:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746504:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757463.746508:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757463.746513:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757463.746520:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757463.746524:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757463.746530:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659650176:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757463.746538:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746543:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757463.746546:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757463.746551:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757463.746559:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757463.746564:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757463.746568:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757463.746574:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757463.746578:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757463.746583:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757463.746586:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746589:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757463.746591:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757463.746594:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757463.746598:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757463.746600:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757463.746602:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757463.746605:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757463.746607:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757463.746610:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757463.746612:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746615:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757463.746617:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757463.746619:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757463.746622:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757463.746624:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757463.746627:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757463.746629:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757463.746632:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757463.746635:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757463.746637:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746641:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757463.746643:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757463.746645:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757463.746649:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757463.746651:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757463.746653:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757463.746656:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757463.746659:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757463.746661:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757463.746664:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746668:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757463.746680:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757463.746684:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757463.746689:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757463.746693:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757464.321761:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757464.321781:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757464.321786:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757464.321798:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757464.321806:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757464.321815:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757464.321820:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757464.321825:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757464.321831:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757464.513311:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757464.513320:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757464.513329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757464.513331:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757464.513422:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757464.513429:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757464.513434:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757465.345258:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757465.345265:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757465.345267:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757465.345272:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757465.345276:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757465.345279:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757465.345282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757465.345284:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757465.345287:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757465.537340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757465.537351:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757465.537361:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757465.537365:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757465.537402:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757465.537407:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757465.537412:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757465.793372:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757465.793386:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757465.793390:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757465.793395:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757465.793403:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757465.793416:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10520 00000100:00080000:2.0:1787757465.793432:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757465.793437:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757465.793441:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757465.793448:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757465.793453:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757465.793458:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757465.793464:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757465.793468:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757465.793471:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757465.793480:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757465.793485:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757465.793491:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757465.793497:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757465.793500:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.793504:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757465.793507:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757465.793519:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757465.793521:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.793525:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757465.793528:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757465.793534:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.793548:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757465.793561:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659650688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757465.793575:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757465.793580:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.793605:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757465.793612:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757465.793618:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757465.793621:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757465.793625:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757465.793633:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757465.793640:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10520 00000100:00080000:2.0:1787757465.793649:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757465.793655:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757465.793661:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757465.793661:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757465.793671:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757465.793675:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787757465.793676:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757465.793685:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:2.0:1787757465.793686:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757465.793694:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000020:00000001:2.0:1787757465.793696:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757465.793702:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.793705:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757465.793713:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757465.793716:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:2.0:1787757465.793726:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757465.793727:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757465.793737:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.793740:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757465.793747:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:1.0:1787757465.793758:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659650688:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757465.793760:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757465.793772:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.793776:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757465.793779:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757465.793787:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:2.0:1787757465.793788:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757465.793796:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757465.793799:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.793807:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757465.793809:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.793820:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757465.793823:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757465.793830:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.793834:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757465.793838:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757465.793843:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.793851:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757465.793857:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.793859:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:1.0:1787757465.793871:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:2.0:1787757465.793878:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659650816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757465.793899:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757465.793905:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.793914:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757465.793920:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757465.793927:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757465.793931:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757465.793936:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757465.793940:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757465.793948:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757465.793959:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10520 00000100:00080000:2.0:1787757465.793970:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757465.794006:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757465.794006:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757465.794016:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757465.794020:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:2.0:1787757465.794023:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757465.794028:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757465.794030:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757465.794041:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:2.0:1787757465.794041:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757465.794050:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757465.794055:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.794057:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757465.794062:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757465.794069:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305193665 00000100:00080000:2.0:1787757465.794077:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757465.794086:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757465.794091:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757465.794093:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757465.794099:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757465.794107:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305193665 02000000:00000001:2.0:1787757465.794109:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.794119:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757465.794125:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00100000:2.0:1787757465.794126:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757465.794137:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794139:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757465.794142:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794153:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757465.794160:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757465.794162:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757465.794166:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.794174:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757465.794179:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757465.794181:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757465/real 1787757465] req@ffff8ffb47a39500 x1874588659650688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757465.794260:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757465.794263:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:2.0:1787757465.794269:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794273:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757465.794281:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757465.794285:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757465.794287:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794291:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757465.794298:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757465.794306:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659650944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757465.794316:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659650688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757465.794368:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.794380:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757465.794381:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757465.794391:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794393:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.794401:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757465.794403:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794405:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757465.794414:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757465.794415:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757465.794423:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757465.794427:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757465.794432:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:2.0:1787757465.794435:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757465.794444:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757465.794450:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757465.794455:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757465.794461:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00080000:2.0:1787757465.794468:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10520 00000100:00100000:1.0:1787757465.794479:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659650688:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787757465.794483:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757465.794493:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.794497:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757465.794503:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:2.0:1787757465.794507:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787757465.794511:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:2.0:1787757465.794520:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757465.794523:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757465.794532:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757465.794533:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757465.794538:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757465.794541:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000020:00000001:2.0:1787757465.794546:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:1.0:1787757465.794551:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000020:00000001:2.0:1787757465.794555:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:1.0:1787757465.794559:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757465.794564:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757465.794569:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757465.794572:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757465.794576:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757465.794587:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757465.794589:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757465.794600:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757465.794605:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757465.794608:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:2.0:1787757465.794613:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757465.794616:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757465.794622:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794626:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757465.794630:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757465.794634:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:2.0:1787757465.794638:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757465.794642:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:2.0:1787757465.794647:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794650:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757465.794656:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794658:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.794668:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757465.794672:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757465.794673:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757465.794680:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.794683:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.794690:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757465.794698:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757465.794700:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650816:192.168.201.118@tcp:38:kworker.0 02000000:00000001:2.0:1787757465.794705:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794716:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757465.794724:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.794725:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757465.794734:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.794741:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:2.0:1787757465.794745:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659651072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757465.794751:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757465.794772:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757465.794773:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757465.794780:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794782:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:2.0:1787757465.794792:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794796:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757465.794834:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757465.794846:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757465.794852:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757465.794855:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794860:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757465.794866:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794875:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305192974 00000100:00000001:1.0:1787757465.794881:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305193665 00000100:00000001:1.0:1787757465.794886:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305192974 00000100:00000001:1.0:1787757465.794890:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305193665 00000100:00000001:1.0:1787757465.794894:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757465.794899:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.794903:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757465.794914:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757465/real 1787757465] req@ffff8ffb47a39880 x1874588659650816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757465.794935:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757465.794938:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757465.794945:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757465.794949:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757465.794952:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.794957:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757465.794960:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757465.794969:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659650816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757465.795031:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795036:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757465.795042:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795045:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757465.795048:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795051:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757465.795075:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757465.795081:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757465.795089:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757465.795109:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757465.795115:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650816:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757465.795122:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757465.795125:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757465.795127:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.795131:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757465.795134:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757465.795138:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757465.795143:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757465.795148:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757465.795152:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757465.795155:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.795158:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757465.795164:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795174:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757465.795177:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757465.795183:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757465.795245:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757465.795266:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757465.795271:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757465.795274:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795278:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757465.795284:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795290:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305192972 00000100:00000001:1.0:1787757465.795296:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305193665 00000100:00000001:1.0:1787757465.795300:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305192972 00000100:00000001:1.0:1787757465.795304:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305193665 00000100:00000001:1.0:1787757465.795309:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757465.795313:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795316:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757465.795325:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757465/real 1787757465] req@ffff8ffb47a3a680 x1874588659650944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757465.795350:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757465.795352:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757465.795358:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757465.795361:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757465.795364:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795367:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757465.795370:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757465.795377:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659650944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757465.795392:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795396:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757465.795400:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795403:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757465.795405:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795409:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757465.795429:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757465.795435:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757465.795441:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757465.795446:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757465.795453:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659650944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757465.795461:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795465:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757465.795467:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757465.795470:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757465.795475:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757465.795478:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757465.795480:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757465.795486:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757465.795488:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757465.795492:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757465.795495:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795498:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757465.795500:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757465.795512:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757465.795516:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757465.795518:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757465.795520:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757465.795532:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757465.795535:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757465.795539:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757465.795541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795545:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795552:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757465.795555:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795559:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757465.795561:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757465.795564:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757465.795567:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757465.795569:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.795573:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757465.795577:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757465.795581:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757465.795588:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757465.795594:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757465.795598:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757465.795600:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757465.795604:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757465.795611:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795621:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757465.795624:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757465.795629:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757465.795655:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757465.795664:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757465.795669:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757465.795672:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795675:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757465.795680:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795686:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305192972 00000100:00000001:1.0:1787757465.795691:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305193665 00000100:00000001:1.0:1787757465.795695:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305192972 00000100:00000001:1.0:1787757465.795699:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305193665 00000100:00000001:1.0:1787757465.795703:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757465.795708:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795711:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757465.795719:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757465/real 1787757465] req@ffff8ffb47a38a80 x1874588659651072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757465.795737:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757465.795740:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757465.795744:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757465.795747:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757465.795750:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795754:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757465.795756:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757465.795763:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659651072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757465.795772:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795775:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757465.795778:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795779:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757465.795782:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795784:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757465.795797:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757465.795800:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757465.795804:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757465.795807:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757465.795811:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757465.795814:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795817:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757465.795818:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757465.795820:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757465.795823:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757465.795825:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757465.795826:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757465.795829:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757465.795830:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757465.795834:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757465.795836:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795838:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757465.795842:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757465.795844:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757465.795846:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757465.795848:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757466.369348:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757466.369360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757466.369364:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757466.369371:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757466.369376:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757466.369381:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757466.369385:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757466.369389:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757466.369393:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757466.561319:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757466.561328:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757466.561336:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757466.561338:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757466.561394:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757466.561402:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757466.561408:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757467.393319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757467.393331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757467.393335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757467.393344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757467.393350:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757467.393356:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757467.393360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757467.393365:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757467.393369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757467.585325:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757467.585338:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757467.585347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757467.585349:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757467.585468:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757467.585476:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757467.585485:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757468.417369:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757468.417382:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757468.417386:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757468.417394:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757468.417400:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757468.417425:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757468.417432:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757468.417436:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757468.417441:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757468.609375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757468.609387:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757468.609397:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757468.609400:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757468.609523:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757468.609529:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757468.609535:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757469.441413:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757469.441427:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757469.441430:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757469.441439:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757469.441444:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757469.441450:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757469.441454:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757469.441458:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757469.441462:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757469.633360:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757469.633371:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757469.633380:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757469.633382:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757469.633414:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757469.633418:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757469.633423:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757470.465278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757470.465294:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757470.465296:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757470.465302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757470.465306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757470.465311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757470.465313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757470.465317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757470.465319:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757470.657297:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757470.657303:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757470.657320:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757470.657321:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757470.657362:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757470.657367:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757470.657370:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757470.913321:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757470.913335:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757470.913339:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757470.913344:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757470.913350:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757470.913360:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10525 00000100:00080000:3.0:1787757470.913376:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757470.913381:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757470.913383:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757470.913389:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757470.913392:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757470.913397:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757470.913399:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757470.913402:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757470.913403:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757470.913409:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757470.913423:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913426:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757470.913430:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757470.913434:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913437:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757470.913439:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913454:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757470.913456:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913460:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757470.913463:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757470.913469:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913497:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757470.913509:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659651200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757470.913522:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757470.913527:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913547:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757470.913550:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757470.913554:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757470.913556:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757470.913558:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757470.913561:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757470.913564:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10525 00000100:00080000:3.0:1787757470.913567:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757470.913569:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757470.913570:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757470.913572:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757470.913573:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757470.913575:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757470.913578:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757470.913580:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757470.913582:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757470.913587:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757470.913590:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913592:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757470.913599:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757470.913602:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913605:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757470.913610:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757470.913615:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.913616:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757470.913618:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913627:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757470.913629:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757470.913629:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913635:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757470.913637:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757470.913638:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757470.913640:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.913647:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757470.913649:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.913654:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757470.913658:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659651328/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757470.913665:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757470.913670:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757470.913673:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757470.913674:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913683:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757470.913696:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757470.913696:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757470.913705:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757470.913707:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757470.913711:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757470.913713:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659651200:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757470.913715:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757470.913723:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10525 00000100:00000001:1.0:1787757470.913730:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757470.913730:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757470.913751:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757470.913753:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757470.913754:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:1.0:1787757470.913755:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787757470.913756:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757470.913760:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787757470.913761:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757470.913762:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757470.913766:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757470.913767:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757470.913768:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757470.913771:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757470.913774:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913775:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757470.913777:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757470.913778:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913778:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913783:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757470.913785:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757470.913787:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913788:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913791:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757470.913792:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913793:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757470.913794:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757470.913795:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.913817:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757470.913818:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.913822:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00100000:3.0:1787757470.913831:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659651456/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757470.913834:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757470.913846:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757470.913849:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913854:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757470.913857:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757470.913861:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757470.913863:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757470.913865:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757470.913869:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757470.913872:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10525 00000100:00080000:3.0:1787757470.913878:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757470.913908:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757470.913910:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757470.913912:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757470.913914:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757470.913933:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757470.913935:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757470.913937:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757470.913937:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757470.913940:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757470.913941:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913942:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757470.913944:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757470.913945:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913947:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757470.913950:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757470.913951:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:3.0:1787757470.913953:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913956:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757470.913959:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757470.913961:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913962:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757470.913963:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757470.913965:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.913966:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757470.913969:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.913971:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757470.913975:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757470.913976:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659651584/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757470.913977:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757470.913984:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757470.913985:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757470.913988:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.913991:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305198785 00000100:00000001:3.0:1787757470.913992:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914006:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757470.914012:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305198785 00000100:00000001:1.0:1787757470.914016:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757470.914021:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757470.914024:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914027:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757470.914033:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757470/real 1787757470] req@ffff8ffb45c32a00 x1874588659651200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757470.914042:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757470.914043:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757470.914046:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757470.914048:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757470.914049:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914051:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757470.914053:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757470.914056:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659651200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757470.914067:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914069:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757470.914072:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914073:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757470.914074:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914076:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757470.914077:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757470.914079:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757470.914082:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757470.914084:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757470.914086:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659651200:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757470.914090:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914092:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757470.914093:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757470.914095:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757470.914099:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757470.914101:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757470.914103:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757470.914107:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757470.914120:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757470.914140:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757470.914141:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914144:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914154:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757470.914157:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914160:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757470.914162:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757470.914163:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757470.914165:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757470.914167:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757470.914170:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757470.914172:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757470.914174:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757470.914178:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651328:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757470.914182:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757470.914184:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757470.914186:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757470.914188:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757470.914191:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914198:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757470.914200:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757470.914206:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757470.914256:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757470.914262:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757470.914264:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757470.914266:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914268:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757470.914271:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914276:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305198094 00000100:00000001:1.0:1787757470.914280:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305198785 00000100:00000001:1.0:1787757470.914283:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305198094 00000100:00000001:1.0:1787757470.914285:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305198785 00000100:00000001:1.0:1787757470.914288:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757470.914290:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914292:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757470.914296:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757470/real 1787757470] req@ffff8ffb45c30700 x1874588659651328/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757470.914304:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757470.914306:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757470.914308:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757470.914310:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757470.914312:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914313:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757470.914315:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757470.914319:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659651328/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757470.914333:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914335:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757470.914338:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914339:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757470.914341:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914342:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757470.914363:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757470.914365:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757470.914368:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757470.914370:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757470.914373:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651328:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757470.914377:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757470.914379:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757470.914381:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757470.914383:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757470.914385:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757470.914387:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757470.914390:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757470.914392:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757470.914394:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757470.914396:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757470.914397:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757470.914400:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914406:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757470.914407:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787757470.914411:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757470.914429:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757470.914433:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757470.914435:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757470.914437:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914439:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757470.914442:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914445:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305198091 00000100:00000001:1.0:1787757470.914447:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305198785 00000100:00000001:1.0:1787757470.914450:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305198091 00000100:00000001:1.0:1787757470.914452:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305198785 00000100:00000001:1.0:1787757470.914454:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757470.914456:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914458:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757470.914461:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757470/real 1787757470] req@ffff8ffb45c30000 x1874588659651456/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757470.914469:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757470.914470:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757470.914471:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757470.914473:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757470.914474:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914476:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757470.914477:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757470.914480:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659651456/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757470.914496:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914499:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757470.914501:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914503:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757470.914504:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914506:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757470.914523:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757470.914525:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757470.914528:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757470.914530:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757470.914534:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757470.914538:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757470.914540:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757470.914541:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757470.914543:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757470.914545:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757470.914547:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757470.914549:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757470.914552:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757470.914554:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757470.914555:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757470.914557:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757470.914560:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914564:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757470.914566:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757470.914571:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757470.914579:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757470.914583:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757470.914585:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757470.914587:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914589:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757470.914591:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914594:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305198091 00000100:00000001:1.0:1787757470.914597:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305198785 00000100:00000001:1.0:1787757470.914599:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305198091 00000100:00000001:1.0:1787757470.914602:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305198785 00000100:00000001:1.0:1787757470.914604:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757470.914606:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914608:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757470.914611:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757470/real 1787757470] req@ffff8ffb45c30e00 x1874588659651584/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757470.914626:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757470.914627:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757470.914629:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757470.914631:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757470.914632:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914634:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757470.914635:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757470.914638:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659651584/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757470.914645:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914646:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757470.914649:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914650:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757470.914651:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914653:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757470.914669:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757470.914672:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757470.914674:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757470.914676:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757470.914679:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757470.914683:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914685:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757470.914687:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757470.914689:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757470.914691:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757470.914693:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757470.914694:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757470.914696:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757470.914698:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757470.914712:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757470.914714:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914717:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757470.914719:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757470.914721:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757470.914725:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757470.914727:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757470.914729:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757470.914732:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757470.914734:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757470.914736:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757470.914738:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914741:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757470.914743:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757470.914744:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757470.914747:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757470.914748:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757470.914766:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757470.914768:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757470.914769:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757470.914771:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757470.914772:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914775:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757470.914778:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757470.914780:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757470.914782:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757470.914784:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757471.489367:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757471.489376:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757471.489379:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757471.489386:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757471.489391:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757471.489396:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757471.489400:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757471.489403:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757471.489407:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757471.681345:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757471.681355:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757471.681366:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757471.681372:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757471.681377:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757471.681381:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757471.681386:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757471.681388:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757471.681391:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757471.681396:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757471.681465:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757471.681471:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757471.681475:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757472.513285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757472.513298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757472.513303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757472.513313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757472.513319:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757472.513327:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757472.513332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757472.513336:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757472.513342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757472.705293:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757472.705313:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757472.705323:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757472.705326:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757472.705503:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757472.705518:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757472.705527:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757473.537310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757473.537319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757473.537322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757473.537328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757473.537333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757473.537338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757473.537341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757473.537344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757473.537348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757473.729277:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757473.729287:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757473.729291:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757473.729295:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757473.729301:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757473.729304:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757473.729307:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757473.729311:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757473.729316:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757473.729319:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757473.729342:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757473.729344:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757473.729347:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757474.561332:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757474.561340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757474.561343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757474.561349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757474.561354:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757474.561358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757474.561361:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757474.561364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757474.561367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757474.753347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757474.753357:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757474.753364:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757474.753367:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757474.753439:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757474.753448:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757474.753455:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757475.585275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757475.585281:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757475.585283:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757475.585286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757475.585289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757475.585292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757475.585294:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757475.585296:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757475.585298:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757475.777271:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757475.777279:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757475.777284:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757475.777285:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757475.777398:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757475.777402:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757475.777405:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757476.033292:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757476.033303:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757476.033306:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757476.033309:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757476.033313:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757476.033321:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10530 00000100:00080000:3.0:1787757476.033349:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757476.033353:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757476.033355:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757476.033360:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757476.033363:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757476.033366:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757476.033368:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757476.033370:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757476.033372:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757476.033376:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757476.033378:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757476.033381:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757476.033384:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757476.033387:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033389:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757476.033391:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757476.033404:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757476.033406:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033409:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757476.033412:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757476.033417:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033431:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757476.033457:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659651712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757476.033467:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757476.033471:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033487:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757476.033490:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757476.033493:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757476.033494:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757476.033496:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757476.033498:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757476.033515:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10530 00000100:00080000:3.0:1787757476.033518:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757476.033520:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757476.033521:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757476.033523:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757476.033524:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757476.033526:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757476.033528:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757476.033530:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757476.033530:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757476.033531:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757476.033535:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757476.033536:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757476.033538:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.033541:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757476.033541:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757476.033546:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757476.033549:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757476.033549:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757476.033553:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.033555:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757476.033555:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757476.033560:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:3.0:1787757476.033562:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757476.033563:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757476.033566:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033570:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757476.033571:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659651712:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757476.033574:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757476.033576:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.033577:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757476.033579:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757476.033582:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757476.033583:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.033584:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757476.033587:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033588:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.033592:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787757476.033596:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659651840/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757476.033598:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033611:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757476.033614:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033617:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757476.033619:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757476.033622:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757476.033623:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757476.033624:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757476.033627:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757476.033627:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757476.033640:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00080000:3.0:1787757476.033642:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10530 00000100:00000001:1.0:1787757476.033645:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757476.033646:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757476.033649:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757476.033651:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757476.033653:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757476.033654:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757476.033656:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757476.033657:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757476.033658:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757476.033659:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757476.033662:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757476.033663:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757476.033664:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757476.033666:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757476.033667:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033669:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757476.033671:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757476.033673:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033675:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757476.033679:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757476.033680:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033681:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757476.033682:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757476.033683:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033688:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757476.033692:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1787757476.033696:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659651968/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757476.033702:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757476.033704:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033721:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.033724:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757476.033725:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757476.033729:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757476.033731:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757476.033732:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757476.033735:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757476.033736:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033739:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757476.033740:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757476.033743:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10530 00000100:00000001:1.0:1787757476.033748:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757476.033749:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757476.033753:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757476.033754:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757476.033755:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305203905 00000020:00000001:3.0:1787757476.033756:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757476.033759:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757476.033761:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757476.033763:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757476.033764:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757476.033765:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757476.033768:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787757476.033768:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757476.033770:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757476.033773:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757476.033777:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305203905 02000000:00000001:3.0:1787757476.033777:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757476.033783:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.033786:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757476.033786:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757476.033788:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757476.033811:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033812:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757476.033841:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757476.033842:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757476.033843:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757476.033845:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757476.033845:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757476.033849:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.033850:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.033854:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757476.033855:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757476.033860:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757476/real 1787757476] req@ffff8ffb45c30a80 x1874588659651712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757476.033862:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659652096/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757476.033889:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:1.0:1787757476.033891:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757476.033892:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757476.033896:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757476.033897:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.033903:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757476.033907:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757476.033909:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.033911:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757476.033913:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757476.033918:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659651712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757476.033925:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.033928:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757476.033930:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.033931:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757476.033933:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.033935:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757476.033936:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757476.033938:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757476.033941:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757476.033943:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757476.033946:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659651712:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757476.033950:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.033952:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757476.033953:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757476.033955:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757476.033958:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757476.033960:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757476.033962:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757476.033965:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757476.033966:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757476.033968:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757476.033969:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.033971:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.033977:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757476.033979:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.033980:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757476.033982:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757476.033983:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757476.033984:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757476.033985:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.033987:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757476.033988:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757476.033990:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757476.033992:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651840:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757476.033994:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757476.033995:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757476.033997:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.033998:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757476.034000:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034005:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757476.034006:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757476.034011:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757476.034019:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757476.034023:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757476.034024:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757476.034025:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034027:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757476.034029:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034032:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305203213 00000100:00000001:1.0:1787757476.034034:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305203905 00000100:00000001:1.0:1787757476.034036:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305203213 00000100:00000001:1.0:1787757476.034038:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305203905 00000100:00000001:1.0:1787757476.034039:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757476.034040:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034041:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757476.034044:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757476/real 1787757476] req@ffff8ffb45c33b80 x1874588659651840/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757476.034049:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757476.034050:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757476.034051:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757476.034052:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757476.034053:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034054:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757476.034055:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757476.034058:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659651840/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757476.034063:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034064:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757476.034065:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034066:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757476.034067:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034069:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757476.034083:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757476.034085:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757476.034087:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757476.034089:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757476.034091:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651840:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757476.034093:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757476.034094:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757476.034095:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.034096:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757476.034097:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757476.034098:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757476.034100:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651968:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757476.034102:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757476.034103:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757476.034129:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.034130:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757476.034132:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034137:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757476.034138:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:1.0:1787757476.034141:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757476.034148:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757476.034151:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757476.034153:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757476.034154:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034155:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757476.034157:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034159:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305203210 00000100:00000001:1.0:1787757476.034161:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305203905 00000100:00000001:1.0:1787757476.034162:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305203210 00000100:00000001:1.0:1787757476.034164:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305203905 00000100:00000001:1.0:1787757476.034167:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757476.034169:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034170:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757476.034173:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757476/real 1787757476] req@ffff8ffb45c31c00 x1874588659651968/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757476.034178:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757476.034179:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757476.034180:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757476.034181:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757476.034182:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034183:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757476.034184:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757476.034187:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659651968/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757476.034192:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034194:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757476.034225:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034227:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757476.034227:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034229:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757476.034241:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757476.034243:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757476.034246:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757476.034247:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757476.034250:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659651968:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757476.034281:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757476.034282:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757476.034283:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.034285:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757476.034286:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757476.034288:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757476.034290:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757476.034292:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757476.034294:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757476.034295:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757476.034296:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757476.034298:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034302:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757476.034302:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757476.034306:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757476.034313:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757476.034316:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757476.034335:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757476.034336:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034337:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757476.034339:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034342:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305203211 00000100:00000001:1.0:1787757476.034344:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305203905 00000100:00000001:1.0:1787757476.034346:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305203211 00000100:00000001:1.0:1787757476.034347:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305203905 00000100:00000001:1.0:1787757476.034349:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757476.034350:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034352:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757476.034354:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757476/real 1787757476] req@ffff8ffb45c30e00 x1874588659652096/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757476.034359:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757476.034360:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757476.034362:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757476.034363:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757476.034364:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034365:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757476.034366:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757476.034369:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659652096/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757476.034387:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034389:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757476.034391:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034392:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757476.034393:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034394:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757476.034407:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757476.034408:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757476.034410:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757476.034411:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757476.034413:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757476.034417:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034419:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757476.034421:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757476.034422:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757476.034425:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757476.034426:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757476.034428:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757476.034430:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757476.034432:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757476.034434:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757476.034435:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034438:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757476.034439:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757476.034440:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757476.034444:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757476.034445:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757476.034447:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757476.034449:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757476.034450:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757476.034452:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757476.034453:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034456:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757476.034457:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757476.034459:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757476.034461:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757476.034462:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757476.034464:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757476.034466:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757476.034467:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757476.034469:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757476.034470:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034473:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757476.034496:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757476.034497:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757476.034499:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757476.034500:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757476.609299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757476.609320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757476.609322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757476.609328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757476.609337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757476.609342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757476.609345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757476.609348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757476.609351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757476.801351:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757476.801360:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757476.801367:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757476.801369:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757476.801416:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757476.801423:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757476.801427:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757477.633339:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757477.633347:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757477.633350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757477.633356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757477.633361:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757477.633366:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757477.633369:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757477.633373:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757477.633377:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757477.825312:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757477.825321:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757477.825338:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757477.825340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757477.825362:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757477.825365:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757477.825369:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757478.657242:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757478.657250:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757478.657252:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757478.657259:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757478.657262:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757478.657266:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757478.657268:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757478.657271:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757478.657274:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757478.849334:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757478.849344:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757478.849351:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757478.849353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757478.849415:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757478.849425:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757478.849431:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757479.681328:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757479.681337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757479.681340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757479.681345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757479.681349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757479.681358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757479.681360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757479.681363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757479.681366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757479.873271:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757479.873279:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757479.873286:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757479.873287:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757479.873332:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757479.873339:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757479.873344:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757480.705277:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757480.705285:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757480.705287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757480.705292:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757480.705295:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757480.705299:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757480.705301:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757480.705303:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757480.705306:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757480.897337:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757480.897344:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757480.897351:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757480.897353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757480.897437:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757480.897443:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757480.897447:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757481.153295:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757481.153310:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757481.153314:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757481.153319:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757481.153326:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757481.153337:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10535 00000100:00080000:3.0:1787757481.153352:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757481.153359:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757481.153362:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757481.153368:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757481.153372:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757481.153376:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757481.153380:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757481.153384:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757481.153386:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757481.153393:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757481.153399:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.153404:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757481.153408:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757481.153413:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153417:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757481.153421:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.153437:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757481.153440:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153443:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757481.153446:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757481.153451:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153464:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757481.153475:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659652224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757481.153486:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757481.153491:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153511:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757481.153515:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757481.153519:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757481.153521:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757481.153523:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757481.153526:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757481.153528:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10535 00000100:00080000:3.0:1787757481.153534:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757481.153536:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757481.153538:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757481.153540:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757481.153542:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757481.153545:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757481.153547:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757481.153549:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757481.153550:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757481.153553:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757481.153555:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.153558:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757481.153564:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757481.153566:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153568:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757481.153574:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757481.153578:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153580:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.153585:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757481.153587:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153588:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757481.153590:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757481.153592:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153597:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757481.153603:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659652352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757481.153611:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757481.153613:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757481.153616:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153621:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757481.153626:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757481.153628:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787757481.153633:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757481.153636:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757481.153638:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757481.153643:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787757481.153643:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757481.153649:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153651:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757481.153662:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10535 02000000:00000001:1.0:1787757481.153664:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757481.153674:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757481.153675:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757481.153682:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153688:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757481.153695:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757481.153699:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659652224:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757481.153721:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757481.153723:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757481.153726:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757481.153731:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757481.153743:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757481.153747:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757481.153751:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757481.153757:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757481.153764:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757481.153771:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757481.153771:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757481.153783:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.153785:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757481.153790:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.153791:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757481.153801:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153804:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757481.153808:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757481.153812:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153814:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.153820:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757481.153822:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.153824:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757481.153826:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757481.153829:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 02000000:00000001:3.0:1787757481.153830:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757481.153838:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.153841:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757481.153850:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757481.153862:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659652480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757481.153872:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757481.153874:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153879:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757481.153882:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757481.153887:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757481.153891:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757481.153893:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757481.153896:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757481.153900:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10535 00000100:00080000:3.0:1787757481.153906:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757481.153916:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757481.153918:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757481.153921:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757481.153926:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757481.153928:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757481.153930:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757481.153933:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757481.153935:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757481.153936:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757481.153941:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757481.153943:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757481.153951:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.153953:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757481.153954:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757481.153962:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.153965:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757481.153971:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757481.153974:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757481.153979:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757481.153983:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757481.153984:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757481.153996:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154000:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305209025 00000100:00000001:3.0:1787757481.154001:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757481.154017:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757481.154020:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154021:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757481.154033:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757481.154036:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305209025 02000000:00000001:3.0:1787757481.154038:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757481.154049:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757481.154050:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154056:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757481.154062:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154069:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757481.154069:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757481.154084:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757481/real 1787757481] req@ffff8ffb45c30000 x1874588659652224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757481.154089:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659652608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757481.154149:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757481.154152:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:1.0:1787757481.154154:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757481.154162:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154165:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757481.154170:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154173:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757481.154180:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154224:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757481.154227:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757481.154236:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659652224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757481.154250:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154255:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757481.154261:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154265:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757481.154266:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154271:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757481.154274:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757481.154279:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757481.154285:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757481.154288:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757481.154293:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659652224:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757481.154300:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757481.154303:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757481.154305:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757481.154308:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757481.154312:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757481.154315:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757481.154319:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652352:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757481.154323:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757481.154326:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757481.154328:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757481.154330:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757481.154336:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154343:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757481.154344:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757481.154352:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757481.154368:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757481.154374:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757481.154378:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757481.154379:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154382:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757481.154385:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154390:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305208334 00000100:00000001:1.0:1787757481.154394:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305209025 00000100:00000001:1.0:1787757481.154397:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305208334 00000100:00000001:1.0:1787757481.154401:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305209025 00000100:00000001:1.0:1787757481.154403:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757481.154406:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154408:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757481.154413:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757481/real 1787757481] req@ffff8ffb45c30700 x1874588659652352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757481.154423:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757481.154425:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757481.154427:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757481.154429:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757481.154432:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154434:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757481.154436:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757481.154440:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659652352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757481.154449:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154451:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757481.154454:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154456:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757481.154458:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154460:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757481.154483:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757481.154487:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757481.154493:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757481.154497:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757481.154504:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652352:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757481.154514:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154520:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757481.154523:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757481.154531:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757481.154539:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757481.154545:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757481.154550:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757481.154558:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757481.154562:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757481.154568:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757481.154571:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154574:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757481.154577:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757481.154579:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757481.154584:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757481.154588:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757481.154590:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757481.154593:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757481.154594:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757481.154600:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757481.154603:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154607:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154618:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757481.154621:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154625:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757481.154627:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757481.154629:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757481.154632:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757481.154633:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757481.154637:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757481.154641:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757481.154644:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757481.154649:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757481.154653:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757481.154656:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757481.154658:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757481.154661:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757481.154664:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154673:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757481.154675:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787757481.154681:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757481.154698:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757481.154727:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757481.154731:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757481.154733:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154736:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757481.154739:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154745:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305208331 00000100:00000001:1.0:1787757481.154748:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305209025 00000100:00000001:1.0:1787757481.154751:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305208331 00000100:00000001:1.0:1787757481.154754:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305209025 00000100:00000001:1.0:1787757481.154757:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757481.154759:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154762:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757481.154768:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757481/real 1787757481] req@ffff8ffb45c32a00 x1874588659652480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757481.154779:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757481.154781:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757481.154784:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757481.154786:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757481.154788:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154791:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757481.154793:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757481.154798:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659652480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757481.154806:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154809:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757481.154812:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154814:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757481.154815:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154818:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757481.154836:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757481.154838:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757481.154842:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757481.154844:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757481.154848:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757481.154853:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757481.154855:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757481.154857:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757481.154860:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757481.154863:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757481.154865:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757481.154869:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757481.154873:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757481.154876:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757481.154878:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757481.154881:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757481.154884:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154890:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757481.154892:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757481.154897:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757481.154912:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757481.154917:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757481.154919:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757481.154921:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154923:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757481.154926:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154930:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305208331 00000100:00000001:1.0:1787757481.154933:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305209025 00000100:00000001:1.0:1787757481.154936:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305208331 00000100:00000001:1.0:1787757481.154939:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305209025 00000100:00000001:1.0:1787757481.154942:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757481.154944:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.154946:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757481.154950:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757481/real 1787757481] req@ffff8ffb45c31500 x1874588659652608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757481.154959:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757481.154961:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757481.154963:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757481.154965:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757481.154967:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154969:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757481.154971:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757481.154975:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659652608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757481.154983:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154986:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757481.154989:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154992:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757481.154993:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.154995:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757481.155014:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757481.155017:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757481.155021:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757481.155024:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757481.155028:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757481.155032:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.155034:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757481.155035:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757481.155037:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757481.155040:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757481.155042:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757481.155043:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757481.155046:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757481.155047:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757481.155050:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757481.155051:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.155053:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757481.155055:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757481.155056:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757481.155058:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757481.155060:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757481.155061:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757481.155063:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757481.155065:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757481.155067:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757481.155068:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.155071:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757481.155075:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757481.155076:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757481.155078:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757481.155081:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757481.729257:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757481.729263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757481.729265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757481.729270:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757481.729273:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757481.729276:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757481.729278:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757481.729280:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757481.729282:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757481.921307:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757481.921320:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757481.921327:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757481.921332:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757481.921335:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757481.921339:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757481.921342:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757481.921345:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757481.921347:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757481.921349:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757481.921370:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757481.921373:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757481.921375:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757482.753272:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757482.753278:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757482.753279:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757482.753283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757482.753286:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757482.753290:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757482.753292:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757482.753294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757482.753295:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757482.945282:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757482.945288:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757482.945293:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757482.945294:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757482.945346:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757482.945351:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757482.945355:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757483.777282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757483.777289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757483.777291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757483.777294:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757483.777297:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757483.777300:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757483.777301:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757483.777303:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757483.777305:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757483.969256:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757483.969265:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757483.969268:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757483.969271:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757483.969276:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757483.969279:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757483.969281:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757483.969283:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757483.969286:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757483.969287:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757483.969337:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757483.969340:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757483.969343:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757484.801337:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757484.801348:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757484.801370:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757484.801379:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757484.801384:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757484.801389:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757484.801392:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757484.801396:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757484.801400:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757484.993292:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757484.993311:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757484.993318:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757484.993319:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757484.993414:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757484.993420:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757484.993424:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757485.825380:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757485.825395:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757485.825399:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757485.825409:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757485.825418:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757485.825425:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757485.825430:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757485.825436:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757485.825441:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757486.017376:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757486.017388:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757486.017398:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757486.017401:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757486.017431:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757486.017434:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757486.017438:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757486.273400:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757486.273423:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757486.273429:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757486.273437:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757486.273449:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757486.273465:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10541 00000100:00080000:1.0:1787757486.273489:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757486.273494:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757486.273498:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757486.273503:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757486.273507:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757486.273512:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757486.273517:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757486.273523:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757486.273526:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757486.273535:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757486.273540:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757486.273544:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757486.273549:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757486.273553:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.273558:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757486.273563:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757486.273578:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757486.273581:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.273586:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757486.273591:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757486.273598:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.273649:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757486.273667:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659652736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757486.273684:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757486.273693:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.273730:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757486.273738:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757486.273746:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757486.273750:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757486.273755:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757486.273762:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757486.273768:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787757486.273769:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10541 00000100:00000001:2.0:1787757486.273780:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757486.273785:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:1.0:1787757486.273787:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757486.273791:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757486.273798:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.273800:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757486.273810:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757486.273812:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757486.273817:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787757486.273821:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:2.0:1787757486.273823:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757486.273829:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:2.0:1787757486.273838:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659652736:192.168.201.118@tcp:250:kworker.0 00000020:00000001:1.0:1787757486.273839:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757486.273855:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:1.0:1787757486.273857:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:2.0:1787757486.273862:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757486.273866:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.273873:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787757486.273873:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757486.273886:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757486.273888:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757486.273896:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757486.273913:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757486.273917:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757486.273928:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757486.273932:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.273938:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757486.273944:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00100000:1.0:1787757486.273945:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757486.273949:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000008:00000001:1.0:1787757486.273952:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.273962:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757486.273964:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757486.273980:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757486.273985:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.273989:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757486.273992:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757486.273997:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.274006:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757486.274019:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659652864/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757486.274039:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757486.274056:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787757486.274059:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757486.274063:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757486.274069:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757486.274074:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757486.274077:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757486.274078:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757486.274086:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757486.274092:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757486.274094:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787757486.274105:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10541 00000100:00000001:2.0:1787757486.274125:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757486.274126:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757486.274132:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757486.274142:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757486.274144:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:1.0:1787757486.274158:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757486.274165:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757486.274169:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757486.274170:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757486.274180:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305214145 00000100:00000001:2.0:1787757486.274197:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757486.274209:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305214146 00000020:00000001:1.0:1787757486.274250:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757486.274271:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757486.274276:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757486.274286:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757486.274287:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757486.274295:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757486.274297:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.274300:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.274309:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:1.0:1787757486.274310:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757486.274320:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:2.0:1787757486.274339:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757486/real 1787757486] req@ffff8ffb4a535c00 x1874588659652736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787757486.274340:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.274359:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757486.274372:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:2.0:1787757486.274378:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000008:00000001:1.0:1787757486.274388:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757486.274388:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757486.274401:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.274402:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757486.274410:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757486.274417:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757486.274421:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757486.274424:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:1.0:1787757486.274429:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.274430:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757486.274454:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757486.274461:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:2.0:1787757486.274464:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659652736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787757486.274475:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.274480:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.274489:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757486.274500:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.274500:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.274520:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757486.274524:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.274528:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757486.274533:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1787757486.274537:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659652992/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757486.274540:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757486.274555:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757486.274565:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757486.274587:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659652736:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757486.274590:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757486.274603:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.274605:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757486.274641:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757486.274650:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757486.274650:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757486.274660:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00080000:1.0:1787757486.274662:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 02000000:00000001:2.0:1787757486.274665:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:1.0:1787757486.274667:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757486.274671:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787757486.274672:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:1.0:1787757486.274683:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000020:00000001:2.0:1787757486.274683:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00080000:1.0:1787757486.274712:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10541 00000020:00000001:2.0:1787757486.274715:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00080000:1.0:1787757486.274723:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:2.0:1787757486.274725:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757486.274734:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757486.274736:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757486.274754:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757486.274756:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757486.274763:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787757486.274766:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757486.274774:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787757486.274775:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757486.274786:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757486.274791:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757486.274793:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757486.274796:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757486.274798:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.274800:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757486.274809:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787757486.274817:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757486.274817:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757486.274832:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757486.274834:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.274841:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787757486.274842:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757486.274857:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757486.274859:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757486.274867:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757486.274868:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.274876:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757486.274892:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787757486.274893:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:1.0:1787757486.274908:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757486.274909:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.274924:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757486.274930:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652864:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757486.274936:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757486.274937:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757486.274942:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.274958:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757486.274959:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757486.274962:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757486.274965:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757486.274968:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757486.274971:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757486.274978:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757486.274980:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757486.275000:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659653120/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757486.275001:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757486.275009:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:1.0:1787757486.275027:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757486.275032:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275034:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757486.275043:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787757486.275091:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757486.275100:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757486.275105:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757486.275107:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275109:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757486.275112:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275117:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305213454 00000100:00000001:2.0:1787757486.275137:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305214145 00000100:00000001:2.0:1787757486.275141:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305213454 00000100:00000001:2.0:1787757486.275144:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305214146 00000100:00000001:2.0:1787757486.275147:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757486.275150:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275152:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757486.275160:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757486/real 1787757486] req@ffff8ffb4a534a80 x1874588659652864/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757486.275173:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757486.275175:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757486.275179:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757486.275181:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757486.275182:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275185:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757486.275187:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757486.275191:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659652864/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757486.275200:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275203:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757486.275206:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275208:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757486.275209:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275212:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757486.275306:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757486.275309:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757486.275313:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757486.275315:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757486.275318:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652864:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757486.275324:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757486.275327:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757486.275329:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757486.275331:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757486.275334:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757486.275337:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757486.275341:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757486.275344:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757486.275363:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757486.275366:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757486.275368:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757486.275373:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275381:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757486.275383:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757486.275388:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757486.275407:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757486.275413:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757486.275416:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757486.275418:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275420:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757486.275423:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275428:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305213452 00000100:00000001:2.0:1787757486.275431:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305214145 00000100:00000001:2.0:1787757486.275434:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305213452 00000100:00000001:2.0:1787757486.275437:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305214146 00000100:00000001:2.0:1787757486.275440:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757486.275442:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275444:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757486.275450:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757486/real 1787757486] req@ffff8ffb4a534700 x1874588659652992/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757486.275475:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757486.275476:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757486.275478:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757486.275479:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757486.275480:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275482:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757486.275483:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757486.275486:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659652992/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757486.275497:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275499:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757486.275501:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275502:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757486.275503:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275504:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757486.275515:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757486.275517:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757486.275519:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757486.275520:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757486.275523:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659652992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757486.275526:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275527:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757486.275529:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757486.275530:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757486.275532:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757486.275533:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757486.275534:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757486.275537:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757486.275538:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757486.275540:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757486.275542:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275543:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757486.275544:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757486.275545:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757486.275546:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757486.275547:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757486.275548:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757486.275550:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757486.275550:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757486.275551:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757486.275552:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275554:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275557:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757486.275558:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275560:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757486.275562:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757486.275563:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757486.275564:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757486.275566:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757486.275568:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757486.275569:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757486.275570:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757486.275573:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757486.275576:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757486.275578:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757486.275579:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757486.275580:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757486.275582:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275586:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757486.275587:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787757486.275590:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757486.275599:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757486.275602:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757486.275604:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757486.275605:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275606:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757486.275608:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275633:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305213452 00000100:00000001:2.0:1787757486.275635:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305214145 00000100:00000001:2.0:1787757486.275637:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305213452 00000100:00000001:2.0:1787757486.275639:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305214146 00000100:00000001:2.0:1787757486.275641:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757486.275642:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275644:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757486.275648:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757486/real 1787757486] req@ffff8ffb4a537b80 x1874588659653120/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757486.275660:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757486.275661:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757486.275664:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757486.275665:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757486.275667:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275668:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757486.275669:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757486.275673:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659653120/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757486.275678:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275680:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757486.275681:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275682:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757486.275683:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275685:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757486.275696:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757486.275698:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757486.275700:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757486.275703:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757486.275705:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757486.275708:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275709:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757486.275710:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757486.275712:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757486.275713:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757486.275714:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757486.275715:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757486.275717:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757486.275718:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757486.275720:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757486.275722:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275723:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757486.275726:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757486.275727:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757486.275729:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757486.275730:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757486.849352:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757486.849364:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757486.849368:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757486.849377:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757486.849383:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757486.849389:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757486.849393:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757486.849397:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757486.849401:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757487.041376:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757487.041390:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757487.041404:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757487.041408:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757487.041490:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757487.041498:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757487.041504:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757487.873360:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757487.873374:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757487.873379:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757487.873390:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757487.873415:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757487.873422:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757487.873428:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757487.873433:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757487.873440:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757488.065334:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757488.065347:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757488.065360:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757488.065363:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757488.065494:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757488.065500:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757488.065505:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757488.897362:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757488.897378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757488.897384:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757488.897396:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757488.897403:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757488.897411:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757488.897416:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757488.897423:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757488.897429:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757489.089347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757489.089358:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757489.089366:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757489.089368:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757489.089546:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757489.089555:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757489.089588:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757489.921363:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757489.921371:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757489.921373:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757489.921380:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757489.921384:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757489.921388:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757489.921391:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757489.921394:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757489.921397:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757490.113298:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757490.113306:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757490.113314:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757490.113316:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757490.113352:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757490.113355:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757490.113359:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757490.945356:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757490.945369:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757490.945373:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757490.945381:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757490.945386:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757490.945393:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757490.945396:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757490.945400:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757490.945405:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757491.137390:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757491.137400:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757491.137409:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757491.137428:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757491.137433:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757491.137437:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757491.137442:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757491.137445:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757491.137449:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757491.137452:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757491.137564:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757491.137573:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757491.137579:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757491.393263:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757491.393274:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757491.393276:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757491.393280:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757491.393285:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757491.393292:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10546 00000100:00080000:1.0:1787757491.393303:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757491.393307:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757491.393309:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757491.393313:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757491.393316:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757491.393319:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757491.393322:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757491.393324:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757491.393326:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757491.393330:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757491.393333:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757491.393335:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757491.393339:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757491.393342:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393344:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757491.393346:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757491.393358:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757491.393360:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393363:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757491.393366:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757491.393370:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393382:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757491.393391:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659653248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757491.393402:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757491.393407:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393423:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757491.393427:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757491.393430:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757491.393432:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757491.393433:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757491.393436:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757491.393438:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10546 00000100:00080000:1.0:1787757491.393441:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757491.393443:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757491.393444:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757491.393446:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757491.393447:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757491.393449:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757491.393452:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757491.393454:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757491.393455:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757491.393458:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757491.393461:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757491.393463:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757491.393469:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757491.393473:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393476:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757491.393483:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757491.393488:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393491:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757491.393492:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757491.393498:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757491.393500:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393503:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757491.393505:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757491.393511:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787757491.393512:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393536:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757491.393541:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393542:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757491.393547:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757491.393556:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659653376/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757491.393557:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757491.393563:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757491.393573:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393584:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:2.0:1787757491.393587:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659653248:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757491.393589:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393596:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393599:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757491.393602:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757491.393607:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757491.393609:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:2.0:1787757491.393609:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:1.0:1787757491.393615:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787757491.393616:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393618:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:2.0:1787757491.393624:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:1.0:1787757491.393627:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10546 00000100:00080000:1.0:1787757491.393632:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:2.0:1787757491.393634:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393635:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757491.393642:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757491.393644:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757491.393646:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757491.393648:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757491.393651:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757491.393652:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757491.393654:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757491.393657:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757491.393658:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757491.393659:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393664:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 02000000:00000001:1.0:1787757491.393665:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757491.393670:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757491.393674:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787757491.393675:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393679:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757491.393683:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757491.393686:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393688:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757491.393694:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757491.393696:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393698:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757491.393700:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757491.393701:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393707:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757491.393713:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659653504/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757491.393720:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757491.393722:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393726:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787757491.393730:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787757491.393731:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757491.393738:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757491.393739:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757491.393741:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757491.393744:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757491.393749:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787757491.393750:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10546 00000100:00080000:1.0:1787757491.393758:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757491.393759:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757491.393760:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757491.393783:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757491.393785:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757491.393788:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757491.393791:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757491.393793:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757491.393796:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757491.393798:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757491.393799:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757491.393803:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757491.393807:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393813:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787757491.393814:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757491.393824:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757491.393826:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393829:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757491.393832:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787757491.393833:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757491.393843:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393848:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305219265 00000100:00000001:1.0:1787757491.393849:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757491.393857:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757491.393861:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393863:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757491.393865:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757491.393867:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393872:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757491.393876:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393887:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305219265 00000100:00100000:1.0:1787757491.393888:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659653632/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757491.393906:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757491.393906:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757491.393913:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757491.393919:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393945:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757491.393954:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.393960:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757491.393976:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757491/real 1787757491] req@ffff8ffb4a536300 x1874588659653248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757491.394001:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757491.394005:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757491.394013:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757491.394017:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757491.394021:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394026:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757491.394030:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757491.394039:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659653248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757491.394057:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394063:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757491.394069:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394073:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757491.394076:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394080:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757491.394084:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757491.394090:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757491.394098:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757491.394104:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757491.394112:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659653248:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757491.394122:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394126:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757491.394129:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757491.394150:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757491.394154:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757491.394157:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757491.394160:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757491.394165:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757491.394167:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757491.394171:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757491.394174:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394177:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394187:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757491.394190:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394193:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757491.394195:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757491.394230:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757491.394233:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757491.394235:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757491.394239:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757491.394242:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757491.394245:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757491.394251:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653376:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757491.394257:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757491.394260:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757491.394262:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757491.394264:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757491.394269:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394279:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757491.394281:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787757491.394287:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757491.394307:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757491.394314:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757491.394318:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757491.394321:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394323:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757491.394327:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394333:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305218574 00000100:00000001:2.0:1787757491.394337:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305219265 00000100:00000001:2.0:1787757491.394340:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305218574 00000100:00000001:2.0:1787757491.394343:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305219265 00000100:00000001:2.0:1787757491.394346:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757491.394349:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394351:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757491.394359:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757491/real 1787757491] req@ffff8ffb4a537800 x1874588659653376/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757491.394382:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757491.394385:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757491.394389:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757491.394391:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757491.394393:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394396:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757491.394398:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757491.394405:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659653376/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757491.394417:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394436:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757491.394442:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394445:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757491.394448:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394451:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757491.394475:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757491.394482:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757491.394489:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757491.394493:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757491.394500:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653376:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757491.394509:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757491.394512:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757491.394515:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757491.394564:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757491.394567:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757491.394570:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757491.394575:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757491.394580:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757491.394583:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757491.394586:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757491.394589:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757491.394594:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394602:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757491.394604:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787757491.394610:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757491.394629:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757491.394636:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757491.394639:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757491.394642:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394644:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757491.394649:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394654:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305218571 00000100:00000001:2.0:1787757491.394658:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305219265 00000100:00000001:2.0:1787757491.394661:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305218571 00000100:00000001:2.0:1787757491.394664:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305219265 00000100:00000001:2.0:1787757491.394667:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757491.394670:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394673:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757491.394679:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757491/real 1787757491] req@ffff8ffb4a537b80 x1874588659653504/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757491.394693:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757491.394695:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757491.394698:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757491.394701:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757491.394703:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394706:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757491.394707:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757491.394714:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659653504/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757491.394724:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394727:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757491.394731:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394734:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757491.394736:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394738:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757491.394755:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757491.394759:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757491.394763:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757491.394767:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757491.394772:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757491.394777:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757491.394779:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757491.394782:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757491.394785:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757491.394787:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757491.394791:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757491.394795:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757491.394800:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757491.394803:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757491.394805:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757491.394808:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757491.394812:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394819:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757491.394822:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757491.394826:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757491.394843:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757491.394849:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757491.394852:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757491.394855:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394857:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757491.394861:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394866:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305218571 00000100:00000001:2.0:1787757491.394870:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305219265 00000100:00000001:2.0:1787757491.394874:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305218571 00000100:00000001:2.0:1787757491.394877:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305219265 00000100:00000001:2.0:1787757491.394880:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757491.394883:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.394886:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757491.394892:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757491/real 1787757491] req@ffff8ffb4a534700 x1874588659653632/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757491.394905:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757491.394907:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757491.394911:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757491.394913:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757491.394915:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394918:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757491.394920:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757491.394926:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659653632/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757491.394937:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394940:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757491.394944:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394946:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757491.394948:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394951:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757491.394967:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757491.394971:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757491.394976:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757491.394980:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757491.394984:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757491.394990:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.394993:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757491.394996:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757491.394998:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757491.395002:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757491.395004:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757491.395006:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757491.395010:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757491.395012:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757491.395015:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757491.395017:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.395019:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757491.395021:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757491.395023:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757491.395027:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757491.395029:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757491.395030:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757491.395032:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757491.395034:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757491.395036:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757491.395038:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.395040:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757491.395042:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757491.395043:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757491.395046:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757491.395048:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757491.395049:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757491.395052:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757491.395070:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757491.395074:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757491.395076:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.395080:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757491.395087:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757491.395090:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757491.395094:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757491.395098:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757491.969418:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757491.969434:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757491.969439:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757491.969452:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757491.969460:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757491.969469:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757491.969474:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757491.969481:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757491.969487:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757492.161418:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757492.161429:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757492.161439:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757492.161441:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757492.161545:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757492.161554:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757492.161561:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757492.993431:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757492.993448:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757492.993452:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757492.993464:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757492.993472:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757492.993481:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757492.993486:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757492.993515:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757492.993522:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757493.185350:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757493.185364:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757493.185370:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757493.185375:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757493.185383:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757493.185387:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757493.185390:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757493.185393:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757493.185399:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757493.185401:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757493.185545:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757493.185556:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757493.185564:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757494.017303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757494.017319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757494.017324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757494.017335:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757494.017343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757494.017352:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757494.017358:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757494.017364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757494.017370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757494.209397:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757494.209408:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757494.209417:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757494.209419:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757494.209467:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757494.209498:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757494.209504:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757495.041381:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757495.041395:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757495.041400:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757495.041410:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757495.041416:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757495.041423:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757495.041427:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757495.041431:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757495.041435:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757495.233365:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757495.233376:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757495.233385:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757495.233387:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757495.233474:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757495.233482:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757495.233490:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757496.065330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757496.065340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757496.065343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757496.065350:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757496.065355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757496.065359:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757496.065362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757496.065366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757496.065369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757496.257361:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757496.257369:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757496.257379:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757496.257381:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757496.257498:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757496.257506:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757496.257513:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757496.513308:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757496.513339:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757496.513343:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757496.513349:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757496.513356:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757496.513369:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10551 00000100:00080000:1.0:1787757496.513386:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757496.513390:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757496.513394:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757496.513400:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757496.513404:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757496.513409:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757496.513412:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757496.513415:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757496.513417:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757496.513426:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757496.513482:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757496.513488:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757496.513496:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757496.513502:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.513507:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757496.513513:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757496.513538:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757496.513542:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.513549:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757496.513553:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757496.513561:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.513582:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757496.513598:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659653760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757496.513614:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757496.513622:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.513649:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757496.513653:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757496.513658:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757496.513661:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757496.513663:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757496.513667:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757496.513671:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10551 00000100:00080000:1.0:1787757496.513677:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757496.513682:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757496.513684:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757496.513689:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757496.513692:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757496.513695:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757496.513699:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757496.513702:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757496.513704:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757496.513709:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757496.513714:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.513714:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787757496.513728:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757496.513730:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757496.513736:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787757496.513739:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757496.513740:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757496.513749:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757496.513751:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.513768:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:2.0:1787757496.513772:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00100000:1.0:1787757496.513778:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787757496.513778:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:1.0:1787757496.513799:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.513800:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.513817:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757496.513828:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659653760:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757496.513829:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757496.513834:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.513849:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757496.513851:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757496.513854:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757496.513860:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.513860:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757496.513867:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.513873:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757496.513874:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.513887:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757496.513889:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659653888/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757496.513904:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757496.513908:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.513913:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757496.513915:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.513917:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00080000:1.0:1787757496.513924:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757496.513928:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787757496.513940:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757496.513946:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757496.513950:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757496.513955:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757496.513960:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10551 00000100:00080000:1.0:1787757496.513965:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757496.513968:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757496.513970:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757496.513974:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757496.513976:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757496.513980:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757496.513983:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757496.513994:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000800:00000100:2.0:1787757496.513995:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757496.514016:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757496.514019:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757496.514024:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757496.514029:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514034:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:1.0:1787757496.514041:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757496.514045:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.514052:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514058:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305224385 02000000:00000001:1.0:1787757496.514061:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757496.514074:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757496.514082:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:1.0:1787757496.514083:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.514094:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757496.514100:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757496.514102:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305224385 00000008:00000001:1.0:1787757496.514111:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.514117:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514120:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787757496.514126:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757496.514130:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.514133:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757496.514136:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757496.514139:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.514147:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757496.514154:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659654016/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757496.514158:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757496.514173:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.514239:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757496.514245:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757496.514246:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757496.514259:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757496.514265:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757496.514277:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787757496.514277:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757496/real 1787757496] req@ffff8ffb4a534a80 x1874588659653760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757496.514288:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757496.514299:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757496.514322:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757496.514334:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10551 00000400:00000001:2.0:1787757496.514336:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757496.514341:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:1.0:1787757496.514344:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757496.514351:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757496.514353:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757496.514362:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757496.514364:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757496.514374:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787757496.514376:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757496.514388:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757496.514389:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:1.0:1787757496.514400:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757496.514402:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787757496.514417:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:2.0:1787757496.514425:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659653760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757496.514451:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757496.514459:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757496.514490:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757496.514502:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757496.514517:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757496.514527:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757496.514531:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757496.514532:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:1.0:1787757496.514554:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514555:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.514577:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757496.514578:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757496.514581:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:1.0:1787757496.514584:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757496.514587:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757496.514593:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00000001:1.0:1787757496.514596:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757496.514602:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787757496.514604:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757496.514622:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00080000:2.0:1787757496.514622:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:1.0:1787757496.514633:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514635:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1787757496.514648:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757496.514658:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:2.0:1787757496.514658:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659653760:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757496.514669:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514674:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757496.514681:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514683:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757496.514692:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757496.514698:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00100000:1.0:1787757496.514705:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659654144/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757496.514708:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757496.514720:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757496.514728:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:1.0:1787757496.514740:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787757496.514741:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757496.514753:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757496.514756:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514764:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757496.514768:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514768:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.514789:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.514801:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757496.514804:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514808:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757496.514811:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757496.514813:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757496.514815:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757496.514818:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.514821:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757496.514825:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757496.514828:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757496.514834:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653888:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757496.514838:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757496.514841:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757496.514843:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.514846:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757496.514850:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514859:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757496.514860:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787757496.514865:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757496.514884:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757496.514890:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757496.514893:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757496.514895:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514897:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757496.514901:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514905:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305223694 00000100:00000001:2.0:1787757496.514908:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305224385 00000100:00000001:2.0:1787757496.514911:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305223694 00000100:00000001:2.0:1787757496.514914:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305224385 00000100:00000001:2.0:1787757496.514916:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757496.514918:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.514920:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757496.514925:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757496/real 1787757496] req@ffff8ffb4a535c00 x1874588659653888/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757496.514934:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757496.514935:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757496.514938:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757496.514940:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757496.514942:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.514943:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757496.514945:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757496.514949:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659653888/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757496.514964:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.514966:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757496.514969:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.514971:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757496.514972:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.514974:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757496.514987:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757496.514990:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757496.514993:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757496.514996:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757496.514999:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659653888:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757496.515004:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757496.515006:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757496.515008:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.515010:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757496.515012:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757496.515014:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757496.515017:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757496.515021:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757496.515023:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757496.515024:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.515026:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757496.515029:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515034:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757496.515035:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757496.515040:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757496.515056:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757496.515063:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757496.515067:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757496.515069:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515072:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757496.515077:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515083:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305223691 00000100:00000001:2.0:1787757496.515087:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305224385 00000100:00000001:2.0:1787757496.515091:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305223691 00000100:00000001:2.0:1787757496.515095:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305224385 00000100:00000001:2.0:1787757496.515099:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757496.515102:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515106:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757496.515114:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757496/real 1787757496] req@ffff8ffb4a534700 x1874588659654016/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757496.515129:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757496.515131:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757496.515136:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757496.515139:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757496.515141:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515143:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757496.515144:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757496.515149:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659654016/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757496.515157:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515160:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757496.515162:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515164:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757496.515166:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515167:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757496.515220:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757496.515226:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757496.515232:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757496.515236:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757496.515242:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757496.515249:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757496.515252:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757496.515254:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.515258:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757496.515262:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757496.515265:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757496.515270:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757496.515276:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757496.515279:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757496.515282:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757496.515285:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757496.515291:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515301:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757496.515303:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787757496.515309:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757496.515330:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757496.515338:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757496.515342:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757496.515344:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515347:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757496.515351:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515357:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305223692 00000100:00000001:2.0:1787757496.515361:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305224385 00000100:00000001:2.0:1787757496.515365:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305223692 00000100:00000001:2.0:1787757496.515368:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305224385 00000100:00000001:2.0:1787757496.515372:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757496.515375:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515379:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757496.515385:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757496/real 1787757496] req@ffff8ffb4a537b80 x1874588659654144/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757496.515408:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757496.515410:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757496.515414:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757496.515416:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757496.515419:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515422:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757496.515424:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757496.515453:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659654144/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757496.515465:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515469:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757496.515473:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515476:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757496.515478:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515481:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757496.515499:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757496.515503:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757496.515508:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757496.515512:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757496.515518:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757496.515524:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515527:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757496.515529:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757496.515532:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757496.515536:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757496.515539:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757496.515541:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757496.515545:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757496.515547:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757496.515551:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757496.515553:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515556:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757496.515558:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757496.515561:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757496.515563:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757496.515566:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757496.515567:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757496.515570:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757496.515572:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757496.515575:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757496.515577:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515579:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757496.515581:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757496.515583:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757496.515586:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757496.515588:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757496.515590:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757496.515593:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757496.515595:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757496.515598:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757496.515600:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515603:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757496.515609:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757496.515612:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757496.515615:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757496.515618:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757497.089356:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757497.089365:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757497.089367:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757497.089374:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757497.089379:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757497.089385:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757497.089389:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757497.089392:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757497.089395:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757497.281344:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757497.281356:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757497.281370:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757497.281375:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757497.281470:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757497.281478:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757497.281485:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757498.113268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757498.113275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757498.113277:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757498.113283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757498.113287:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757498.113292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757498.113295:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757498.113298:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757498.113301:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757498.305447:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757498.305455:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757498.305462:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757498.305463:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757498.305482:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757498.305485:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757498.305487:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757499.137281:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757499.137287:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757499.137289:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757499.137293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757499.137296:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757499.137299:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757499.137301:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757499.137303:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757499.137305:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757499.329305:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757499.329311:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757499.329316:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757499.329317:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757499.329433:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757499.329437:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757499.329440:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757500.161290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757500.161316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757500.161320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757500.161328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757500.161334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757500.161341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757500.161347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757500.161354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757500.161359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757500.353353:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757500.353418:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757500.353433:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757500.353436:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757500.353544:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757500.353551:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757500.353556:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757501.185296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757501.185307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757501.185311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757501.185320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757501.185326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757501.185332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757501.185337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757501.185341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757501.185368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757501.377332:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757501.377340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757501.377373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757501.377378:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757501.377382:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757501.377399:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757501.377403:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757501.377406:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757501.377408:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757501.377412:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757501.377495:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757501.377501:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757501.377506:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757501.633398:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757501.633411:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757501.633415:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757501.633420:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757501.633426:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757501.633437:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10556 00000100:00080000:1.0:1787757501.633451:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757501.633456:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757501.633459:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757501.633465:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757501.633468:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757501.633472:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757501.633475:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757501.633477:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757501.633479:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757501.633485:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757501.633489:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.633493:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757501.633498:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757501.633502:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633505:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757501.633508:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.633523:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757501.633525:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633529:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757501.633532:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757501.633538:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633552:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757501.633564:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659654272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757501.633578:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757501.633583:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633603:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757501.633607:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757501.633611:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757501.633613:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757501.633616:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757501.633619:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757501.633622:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10556 00000100:00080000:1.0:1787757501.633626:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757501.633629:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757501.633630:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757501.633633:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757501.633635:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757501.633638:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757501.633641:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757501.633644:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757501.633645:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757501.633649:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757501.633652:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.633655:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757501.633661:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757501.633665:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633667:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757501.633675:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757501.633680:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633683:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.633688:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757501.633690:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633693:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757501.633695:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757501.633697:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.633701:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757501.633703:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.633709:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:1.0:1787757501.633716:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659654400/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757501.633717:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757501.633723:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757501.633729:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633748:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787757501.633750:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757501.633755:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757501.633758:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757501.633763:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757501.633770:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757501.633774:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757501.633775:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757501.633777:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787757501.633779:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633781:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757501.633794:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10556 00000100:00100000:2.0:1787757501.633796:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659654272:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787757501.633809:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757501.633815:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787757501.633816:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757501.633822:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757501.633823:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:1.0:1787757501.633827:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:2.0:1787757501.633829:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757501.633831:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757501.633838:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757501.633840:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:1.0:1787757501.633848:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757501.633854:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633855:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757501.633862:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757501.633869:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757501.633872:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.633874:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757501.633878:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757501.633881:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633891:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757501.633892:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00100000:1.0:1787757501.633894:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757501.633897:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000008:00000001:1.0:1787757501.633902:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.633907:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757501.633908:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.633920:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757501.633922:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633925:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757501.633926:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757501.633929:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633937:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757501.633944:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659654528/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757501.633954:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757501.633956:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.633959:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757501.633964:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787757501.633970:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787757501.633971:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757501.633978:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757501.633982:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757501.633985:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757501.633990:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10556 00000100:00080000:1.0:1787757501.633997:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757501.634000:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757501.634001:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757501.634008:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757501.634013:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:1.0:1787757501.634014:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757501.634022:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757501.634024:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757501.634036:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757501.634042:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:1.0:1787757501.634043:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757501.634050:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757501.634057:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.634064:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757501.634068:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305229505 00000100:00000001:2.0:1787757501.634086:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757501.634096:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305229505 00000100:00000001:2.0:1787757501.634102:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787757501.634110:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757501.634114:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.634116:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757501.634120:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757501.634121:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757501.634130:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.634132:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.634141:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757501.634143:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:1.0:1787757501.634146:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757501.634152:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757501.634158:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757501/real 1787757501] req@ffff8ffb4a537800 x1874588659654272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757501.634159:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757501.634173:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757501.634178:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.634233:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757501.634236:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757501.634239:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757501.634247:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757501.634248:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757501.634254:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:1.0:1787757501.634259:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659654656/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757501.634263:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757501.634273:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757501.634275:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757501.634280:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757501.634282:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634287:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757501.634290:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634293:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757501.634306:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659654272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757501.634317:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634323:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757501.634329:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634331:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757501.634334:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634361:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757501.634365:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757501.634370:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757501.634386:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757501.634389:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757501.634394:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659654272:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757501.634400:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634403:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757501.634405:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757501.634408:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757501.634413:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757501.634415:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757501.634418:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757501.634424:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757501.634426:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757501.634429:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757501.634431:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634434:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634443:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757501.634445:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634448:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757501.634450:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757501.634452:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757501.634454:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757501.634456:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757501.634459:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757501.634461:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757501.634464:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757501.634469:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654400:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757501.634473:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757501.634476:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757501.634478:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757501.634480:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757501.634484:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634493:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757501.634495:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787757501.634501:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757501.634520:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757501.634526:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757501.634529:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757501.634531:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634533:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757501.634537:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634542:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305228814 00000100:00000001:2.0:1787757501.634546:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305229505 00000100:00000001:2.0:1787757501.634549:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305228814 00000100:00000001:2.0:1787757501.634552:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305229505 00000100:00000001:2.0:1787757501.634555:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757501.634557:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634559:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757501.634565:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757501/real 1787757501] req@ffff8ffb4a536300 x1874588659654400/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757501.634575:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757501.634576:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757501.634579:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757501.634581:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757501.634583:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634585:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757501.634587:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757501.634592:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659654400/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757501.634600:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634603:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757501.634606:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634607:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757501.634609:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634612:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757501.634627:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757501.634630:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757501.634634:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757501.634637:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757501.634641:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654400:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757501.634645:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757501.634647:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757501.634649:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757501.634651:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757501.634653:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757501.634656:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757501.634659:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757501.634663:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757501.634665:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757501.634667:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757501.634669:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757501.634673:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634678:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757501.634679:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787757501.634684:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757501.634695:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757501.634700:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757501.634703:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757501.634704:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634706:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757501.634709:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634713:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305228811 00000100:00000001:2.0:1787757501.634716:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305229505 00000100:00000001:2.0:1787757501.634719:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305228811 00000100:00000001:2.0:1787757501.634722:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305229505 00000100:00000001:2.0:1787757501.634726:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757501.634728:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634731:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757501.634736:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757501/real 1787757501] req@ffff8ffb4a537b80 x1874588659654528/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757501.634751:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757501.634753:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757501.634755:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757501.634757:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757501.634758:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634760:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757501.634762:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757501.634767:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659654528/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757501.634776:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634778:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757501.634781:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634783:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757501.634784:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634786:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757501.634798:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757501.634801:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757501.634805:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757501.634807:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757501.634811:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757501.634815:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757501.634816:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757501.634818:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757501.634820:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757501.634823:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757501.634825:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757501.634829:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757501.634833:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757501.634835:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757501.634837:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757501.634839:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757501.634842:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634847:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757501.634849:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757501.634853:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757501.634875:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757501.634880:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757501.634882:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757501.634884:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634886:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757501.634888:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634892:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305228811 00000100:00000001:2.0:1787757501.634895:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305229505 00000100:00000001:2.0:1787757501.634898:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305228811 00000100:00000001:2.0:1787757501.634901:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305229505 00000100:00000001:2.0:1787757501.634903:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757501.634905:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.634907:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757501.634912:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757501/real 1787757501] req@ffff8ffb4a534700 x1874588659654656/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757501.634920:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757501.634922:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757501.634924:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757501.634926:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757501.634927:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634929:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757501.634931:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757501.634935:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659654656/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757501.634952:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634955:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757501.634959:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634961:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757501.634964:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.634967:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757501.634986:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757501.634992:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757501.634998:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757501.635002:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757501.635009:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757501.635017:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.635020:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757501.635023:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757501.635026:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757501.635030:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757501.635033:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757501.635036:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757501.635042:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757501.635044:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757501.635048:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757501.635051:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.635054:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757501.635056:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757501.635059:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757501.635062:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757501.635064:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757501.635078:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757501.635080:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757501.635082:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757501.635084:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757501.635086:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.635088:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757501.635089:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757501.635092:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757501.635095:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757501.635097:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757501.635098:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757501.635100:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757501.635102:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757501.635104:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757501.635105:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.635108:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757501.635112:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757501.635114:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757501.635117:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757501.635119:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757502.209326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757502.209372:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757502.209375:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757502.209381:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757502.209386:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757502.209390:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757502.209394:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757502.209398:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757502.209401:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757502.401375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757502.401384:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757502.401393:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757502.401396:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757502.401432:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757502.401435:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757502.401439:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757503.233418:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757503.233430:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757503.233435:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757503.233443:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757503.233450:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757503.233456:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757503.233461:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757503.233467:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757503.233473:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757503.425355:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757503.425370:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757503.425375:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757503.425381:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757503.425391:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757503.425395:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757503.425400:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757503.425405:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757503.425414:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757503.425417:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757503.425498:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757503.425504:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757503.425509:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757504.257406:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757504.257412:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757504.257414:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757504.257418:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757504.257421:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757504.257424:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757504.257426:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757504.257428:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757504.257430:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757504.449332:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757504.449343:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757504.449352:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757504.449354:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757504.449415:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757504.449420:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757504.449424:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757505.281391:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757505.281400:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757505.281403:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757505.281410:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757505.281414:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757505.281420:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757505.281424:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757505.281427:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757505.281430:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757505.473310:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757505.473320:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757505.473331:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757505.473335:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757505.473407:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757505.473413:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757505.473417:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757506.305336:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757506.305347:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757506.305350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757506.305356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757506.305361:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757506.305366:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757506.305370:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757506.305374:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757506.305378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757506.497275:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757506.497280:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757506.497285:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757506.497286:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757506.497305:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757506.497307:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757506.497309:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757506.753285:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757506.753293:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757506.753295:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757506.753299:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757506.753302:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757506.753309:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10561 00000100:00080000:1.0:1787757506.753320:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757506.753323:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757506.753325:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757506.753329:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757506.753331:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757506.753333:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757506.753335:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757506.753336:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757506.753337:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757506.753341:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757506.753343:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753345:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757506.753347:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757506.753349:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753351:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757506.753352:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753362:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757506.753363:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753366:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757506.753368:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757506.753372:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753382:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757506.753389:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659654784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757506.753396:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757506.753399:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753409:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757506.753411:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757506.753414:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757506.753415:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757506.753416:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757506.753418:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757506.753420:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10561 00000100:00080000:1.0:1787757506.753423:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757506.753424:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757506.753425:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757506.753427:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757506.753428:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757506.753429:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757506.753431:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757506.753432:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757506.753433:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757506.753434:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757506.753436:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753437:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757506.753441:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757506.753442:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753444:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757506.753448:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757506.753450:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753452:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753455:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757506.753455:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753456:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757506.753457:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757506.753459:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753459:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757506.753465:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757506.753466:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753468:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757506.753471:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00100000:1.0:1787757506.753472:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659654912/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757506.753473:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757506.753480:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757506.753484:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787757506.753484:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757506.753488:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757506.753493:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659654784:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757506.753494:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753499:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753500:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757506.753503:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787757506.753506:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:1.0:1787757506.753507:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:2.0:1787757506.753509:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753510:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757506.753514:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787757506.753514:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757506.753518:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:2.0:1787757506.753520:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757506.753521:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10561 00000100:00080000:1.0:1787757506.753525:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757506.753527:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757506.753528:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757506.753530:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757506.753531:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757506.753533:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757506.753534:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757506.753536:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757506.753536:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757506.753540:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:1.0:1787757506.753541:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757506.753547:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787757506.753548:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757506.753552:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753553:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757506.753555:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757506.753557:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753558:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757506.753560:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757506.753561:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753563:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753566:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757506.753566:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753567:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757506.753568:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757506.753570:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753574:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757506.753577:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659655040/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757506.753584:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757506.753588:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757506.753591:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753594:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757506.753596:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757506.753598:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787757506.753599:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757506.753602:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757506.753604:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787757506.753605:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757506.753608:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753609:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757506.753612:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:1.0:1787757506.753613:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10561 00000100:00080000:1.0:1787757506.753619:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757506.753619:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753622:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757506.753624:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757506.753625:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305234625 00000020:00000001:1.0:1787757506.753628:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757506.753630:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757506.753631:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757506.753633:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757506.753634:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757506.753636:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757506.753636:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787757506.753641:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757506.753643:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753645:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757506.753645:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305234625 02000000:00000001:1.0:1787757506.753649:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757506.753651:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787757506.753652:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753655:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757506.753657:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757506.753658:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753659:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753662:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757506.753662:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753664:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757506.753665:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757506.753667:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757506.753669:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757506.753670:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753672:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757506.753675:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757506.753685:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659655168/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757506.753685:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757506/real 1787757506] req@ffff8ffb4a535c00 x1874588659654784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757506.753699:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757506.753703:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757506.753706:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757506.753714:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757506.753717:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757506.753721:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757506.753724:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757506.753726:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753729:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757506.753731:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757506.753737:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659654784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757506.753745:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753747:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757506.753749:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753751:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757506.753751:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753753:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757506.753755:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757506.753757:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757506.753759:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757506.753761:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757506.753764:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659654784:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757506.753768:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753770:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757506.753771:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757506.753773:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757506.753776:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757506.753778:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757506.753779:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757506.753782:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757506.753783:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757506.753785:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757506.753786:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753787:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753793:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757506.753795:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753797:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757506.753798:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757506.753799:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757506.753800:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757506.753801:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757506.753803:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757506.753804:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757506.753805:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757506.753808:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654912:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757506.753810:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757506.753811:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757506.753812:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757506.753814:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757506.753816:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753821:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757506.753822:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787757506.753825:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757506.753835:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757506.753838:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757506.753839:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757506.753840:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753841:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757506.753843:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753846:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305233933 00000100:00000001:2.0:1787757506.753848:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305234625 00000100:00000001:2.0:1787757506.753850:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305233933 00000100:00000001:2.0:1787757506.753851:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305234625 00000100:00000001:2.0:1787757506.753853:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757506.753854:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753855:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757506.753858:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757506/real 1787757506] req@ffff8ffb4a534a80 x1874588659654912/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757506.753863:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757506.753863:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757506.753865:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757506.753866:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757506.753867:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753869:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757506.753870:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757506.753872:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659654912/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757506.753876:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753878:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757506.753879:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753880:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757506.753881:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753882:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757506.753892:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757506.753893:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757506.753895:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757506.753897:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757506.753899:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659654912:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757506.753901:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757506.753902:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757506.753904:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757506.753905:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757506.753906:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757506.753908:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757506.753910:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757506.753911:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757506.753912:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757506.753913:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757506.753914:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757506.753916:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753919:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757506.753920:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757506.753922:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757506.753928:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757506.753930:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757506.753931:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757506.753932:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753933:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757506.753935:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753937:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305233930 00000100:00000001:2.0:1787757506.753939:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305234625 00000100:00000001:2.0:1787757506.753940:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305233930 00000100:00000001:2.0:1787757506.753942:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305234625 00000100:00000001:2.0:1787757506.753943:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757506.753944:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.753945:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757506.753947:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757506/real 1787757506] req@ffff8ffb4a534700 x1874588659655040/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757506.753952:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757506.753953:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757506.753954:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757506.753955:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757506.753955:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753957:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757506.753958:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757506.753960:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659655040/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757506.753965:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753966:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757506.753967:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753968:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757506.753969:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.753970:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757506.753978:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757506.753979:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757506.753981:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757506.753982:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757506.753984:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757506.753986:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757506.753987:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757506.753988:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757506.753989:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757506.753990:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757506.753991:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757506.753993:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757506.753995:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757506.753996:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757506.753997:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757506.753998:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757506.753999:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.754002:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757506.754003:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787757506.754005:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757506.754010:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757506.754013:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757506.754014:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757506.754015:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.754016:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757506.754017:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.754019:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305233930 00000100:00000001:2.0:1787757506.754021:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305234625 00000100:00000001:2.0:1787757506.754022:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305233930 00000100:00000001:2.0:1787757506.754024:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305234625 00000100:00000001:2.0:1787757506.754025:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757506.754026:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.754027:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757506.754030:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757506/real 1787757506] req@ffff8ffb4a537b80 x1874588659655168/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757506.754034:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757506.754035:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757506.754036:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757506.754037:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757506.754037:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754038:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757506.754039:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757506.754041:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659655168/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757506.754045:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754046:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757506.754048:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754049:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757506.754050:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754051:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757506.754059:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757506.754061:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757506.754063:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757506.754065:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757506.754067:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757506.754070:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754071:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757506.754072:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757506.754074:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757506.754076:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757506.754077:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757506.754078:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757506.754081:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757506.754082:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757506.754084:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757506.754086:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754087:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757506.754089:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757506.754090:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757506.754092:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757506.754093:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757506.754094:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757506.754096:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757506.754097:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757506.754099:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757506.754100:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754102:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757506.754104:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757506.754105:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757506.754107:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757506.754108:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757506.754109:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757506.754111:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757506.754112:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757506.754114:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757506.754115:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754117:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757506.754120:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757506.754122:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757506.754124:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757506.754126:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757507.329357:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757507.329364:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757507.329365:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757507.329370:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757507.329373:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757507.329376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757507.329379:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757507.329383:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757507.329385:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757507.521288:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757507.521294:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757507.521301:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757507.521302:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757507.521397:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757507.521405:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757507.521412:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757508.353367:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757508.353378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757508.353381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757508.353389:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757508.353395:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757508.353401:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757508.353404:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757508.353408:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757508.353411:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757508.545402:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757508.545416:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757508.545429:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757508.545431:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757508.545517:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757508.545525:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757508.545530:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757509.377367:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757509.377378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757509.377381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757509.377390:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757509.377396:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757509.377402:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757509.377406:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757509.377410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757509.377414:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757509.569363:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757509.569374:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757509.569382:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757509.569384:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757509.569454:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757509.569480:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757509.569487:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757510.401317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757510.401326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757510.401328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757510.401334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757510.401338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757510.401344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757510.401347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757510.401350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757510.401354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757510.593311:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757510.593319:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757510.593325:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757510.593326:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757510.593352:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757510.593354:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757510.593358:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757511.425315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757511.425321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757511.425322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757511.425326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757511.425329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757511.425332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757511.425334:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757511.425336:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757511.425338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757511.617295:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757511.617302:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757511.617309:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757511.617312:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757511.617315:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757511.617319:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757511.617323:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757511.617325:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757511.617327:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757511.617330:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757511.617382:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757511.617386:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757511.617389:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757511.873319:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757511.873332:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757511.873335:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757511.873339:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757511.873344:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757511.873353:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10566 00000100:00080000:2.0:1787757511.873367:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757511.873370:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757511.873373:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757511.873378:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757511.873381:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757511.873385:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757511.873388:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757511.873391:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757511.873393:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757511.873399:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757511.873420:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757511.873423:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757511.873427:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757511.873431:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.873435:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757511.873439:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757511.873456:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757511.873460:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.873467:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757511.873472:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757511.873479:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.873500:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757511.873518:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659655296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757511.873557:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757511.873566:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.873600:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757511.873607:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757511.873614:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757511.873617:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757511.873621:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757511.873627:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757511.873631:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10566 00000100:00080000:2.0:1787757511.873637:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757511.873640:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757511.873642:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757511.873646:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757511.873647:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757511.873650:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757511.873653:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757511.873656:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757511.873658:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757511.873662:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757511.873665:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757511.873667:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757511.873674:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757511.873677:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.873679:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757511.873703:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757511.873707:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.873710:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757511.873727:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757511.873729:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.873733:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757511.873733:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757511.873738:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757511.873742:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757511.873743:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.873748:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757511.873752:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757511.873755:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.873756:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757511.873768:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:2.0:1787757511.873773:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659655424/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757511.873774:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757511.873784:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757511.873800:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659655296:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757511.873801:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757511.873809:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.873811:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757511.873818:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757511.873819:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757511.873826:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757511.873828:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757511.873832:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:2.0:1787757511.873836:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757511.873841:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:1.0:1787757511.873843:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757511.873844:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757511.873852:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757511.873857:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10566 00000100:00080000:2.0:1787757511.873863:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757511.873866:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757511.873869:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757511.873870:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757511.873874:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000020:00000001:2.0:1787757511.873882:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757511.873883:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787757511.873890:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757511.873897:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757511.873902:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757511.873907:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757511.873909:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757511.873918:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757511.873924:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757511.873927:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757511.873933:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000800:00000100:1.0:1787757511.873935:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:2.0:1787757511.873945:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.873952:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757511.873956:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00100000:2.0:1787757511.873958:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757511.873964:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00000001:2.0:1787757511.873967:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.873970:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.873974:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.873976:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787757511.873985:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757511.873987:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757511.873992:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.873995:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305239745 00000100:00000001:2.0:1787757511.873997:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757511.874003:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757511.874007:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874013:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757511.874016:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874026:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305239745 00000100:00100000:2.0:1787757511.874029:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659655552/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757511.874035:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757511.874045:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757511.874051:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757511.874052:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.874057:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874060:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787757511.874065:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757511.874070:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757511.874073:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757511/real 1787757511] req@ffff8ffb47a38380 x1874588659655296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757511.874075:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757511.874080:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757511.874085:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757511.874095:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757511.874104:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10566 00000400:00000001:1.0:1787757511.874105:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757511.874110:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:2.0:1787757511.874114:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757511.874116:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757511.874124:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757511.874124:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757511.874131:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757511.874132:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757511.874139:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:2.0:1787757511.874140:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757511.874144:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757511.874145:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757511.874153:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757511.874237:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757511.874244:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757511.874247:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757511.874258:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757511.874264:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757511.874267:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757511.874274:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757511.874279:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.874282:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757511.874284:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757511.874288:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.874290:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757511.874297:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:1.0:1787757511.874298:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659655296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757511.874299:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757511.874307:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757511.874312:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757511.874318:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874322:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757511.874328:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874330:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757511.874339:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874345:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:2.0:1787757511.874346:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659655680/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757511.874347:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874355:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757511.874359:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757511.874370:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:2.0:1787757511.874372:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757511.874379:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757511.874381:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757511.874385:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874388:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757511.874397:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659655296:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757511.874421:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874424:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757511.874427:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757511.874431:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757511.874437:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757511.874440:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757511.874444:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757511.874448:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757511.874451:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757511.874455:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757511.874457:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874460:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874468:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757511.874471:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874474:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757511.874476:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757511.874478:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757511.874480:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757511.874482:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757511.874485:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757511.874488:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757511.874491:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757511.874496:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655424:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757511.874500:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757511.874503:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757511.874504:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757511.874507:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757511.874511:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874520:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757511.874522:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:1.0:1787757511.874528:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757511.874546:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757511.874552:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757511.874555:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757511.874557:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874559:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757511.874563:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874568:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305239054 00000100:00000001:1.0:1787757511.874572:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305239745 00000100:00000001:1.0:1787757511.874575:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305239054 00000100:00000001:1.0:1787757511.874578:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305239745 00000100:00000001:1.0:1787757511.874582:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757511.874584:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874586:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757511.874592:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757511/real 1787757511] req@ffff8ffb47a38000 x1874588659655424/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757511.874604:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757511.874605:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757511.874608:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757511.874610:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757511.874612:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874614:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757511.874615:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757511.874620:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659655424/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757511.874629:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874632:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757511.874635:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874637:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757511.874639:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874641:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757511.874655:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757511.874658:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757511.874661:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757511.874681:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757511.874685:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655424:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757511.874690:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757511.874691:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757511.874693:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757511.874695:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757511.874697:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757511.874700:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757511.874704:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757511.874707:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757511.874709:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757511.874711:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757511.874713:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757511.874716:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874723:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757511.874724:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787757511.874729:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757511.874741:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757511.874745:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757511.874748:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757511.874749:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874751:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757511.874754:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874758:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305239051 00000100:00000001:1.0:1787757511.874760:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305239745 00000100:00000001:1.0:1787757511.874763:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305239051 00000100:00000001:1.0:1787757511.874766:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305239745 00000100:00000001:1.0:1787757511.874768:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757511.874771:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874772:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757511.874777:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757511/real 1787757511] req@ffff8ffb47a3b480 x1874588659655552/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757511.874785:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757511.874786:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757511.874788:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757511.874790:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757511.874791:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874793:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757511.874795:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757511.874799:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659655552/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757511.874814:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874817:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757511.874819:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874821:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757511.874823:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874824:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757511.874836:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757511.874839:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757511.874842:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757511.874844:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757511.874848:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757511.874852:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757511.874853:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757511.874855:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757511.874857:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757511.874859:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757511.874862:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757511.874865:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655680:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757511.874868:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757511.874869:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757511.874871:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757511.874873:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757511.874876:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874880:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757511.874882:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787757511.874886:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757511.874895:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757511.874899:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757511.874901:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757511.874903:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874905:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757511.874907:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874911:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305239051 00000100:00000001:1.0:1787757511.874914:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305239745 00000100:00000001:1.0:1787757511.874917:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305239051 00000100:00000001:1.0:1787757511.874919:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305239745 00000100:00000001:1.0:1787757511.874922:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757511.874924:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.874926:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757511.874930:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757511/real 1787757511] req@ffff8ffb47a39180 x1874588659655680/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757511.874938:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757511.874939:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757511.874941:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757511.874943:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757511.874944:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874946:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757511.874948:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757511.874952:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659655680/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757511.874959:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874961:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757511.874964:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874966:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757511.874967:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.874970:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757511.874981:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757511.874984:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757511.874987:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757511.874989:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757511.874993:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655680:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757511.874998:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.875000:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757511.875002:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757511.875003:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757511.875005:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757511.875007:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757511.875008:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757511.875011:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757511.875012:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757511.875014:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757511.875016:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.875018:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757511.875019:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757511.875021:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757511.875024:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757511.875025:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757511.875026:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757511.875029:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757511.875030:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757511.875033:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757511.875035:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.875036:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757511.875038:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757511.875039:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757511.875041:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757511.875042:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757511.875043:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757511.875045:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757511.875046:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757511.875049:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757511.875050:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.875052:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757511.875056:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757511.875058:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757511.875060:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757511.875061:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757512.449289:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757512.449298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757512.449300:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757512.449308:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757512.449312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757512.449317:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757512.449321:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757512.449324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757512.449328:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757512.641384:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757512.641394:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757512.641405:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757512.641409:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757512.641479:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757512.641488:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757512.641496:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757513.473409:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757513.473425:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757513.473429:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757513.473457:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757513.473464:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757513.473473:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757513.473479:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757513.473484:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757513.473490:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757513.665320:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757513.665331:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757513.665333:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757513.665336:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757513.665343:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757513.665345:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757513.665347:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757513.665348:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757513.665352:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757513.665353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757513.665423:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757513.665430:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757513.665434:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757514.497351:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757514.497360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757514.497363:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757514.497370:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757514.497374:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757514.497379:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757514.497382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757514.497386:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757514.497389:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757514.689354:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757514.689365:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757514.689372:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757514.689374:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757514.689407:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757514.689412:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757514.689416:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757515.521242:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757515.521248:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757515.521250:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757515.521254:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757515.521257:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757515.521260:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757515.521262:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757515.521264:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757515.521266:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757515.713256:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757515.713262:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757515.713267:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757515.713269:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757515.713330:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757515.713333:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757515.713336:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757516.545293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757516.545300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757516.545302:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757516.545307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757516.545310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757516.545314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757516.545317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757516.545319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757516.545321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757516.737342:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757516.737349:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757516.737354:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757516.737355:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757516.737461:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757516.737469:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757516.737476:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757516.993363:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757516.993378:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757516.993383:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757516.993388:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757516.993396:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757516.993406:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10571 00000100:00080000:3.0:1787757516.993424:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757516.993429:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757516.993432:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757516.993438:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757516.993442:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757516.993446:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757516.993451:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757516.993454:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757516.993457:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757516.993463:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757516.993467:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757516.993471:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757516.993476:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757516.993480:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993483:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757516.993487:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757516.993506:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757516.993508:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993513:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757516.993517:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757516.993524:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993543:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757516.993556:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659655808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757516.993571:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757516.993578:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993604:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757516.993610:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757516.993614:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757516.993617:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757516.993619:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757516.993623:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757516.993626:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10571 00000100:00080000:3.0:1787757516.993630:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757516.993632:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757516.993634:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757516.993637:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757516.993639:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757516.993642:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757516.993645:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757516.993649:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757516.993651:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757516.993655:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757516.993658:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757516.993660:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757516.993668:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757516.993670:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993673:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757516.993679:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757516.993683:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993687:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757516.993692:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757516.993695:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993697:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757516.993700:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757516.993702:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993709:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757516.993716:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659655936/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757516.993727:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757516.993728:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757516.993734:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.993740:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757516.993741:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757516.993750:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757516.993752:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:3.0:1787757516.993755:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757516.993761:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757516.993762:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787757516.993764:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757516.993772:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.993784:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757516.993788:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757516.993793:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757516.993795:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10571 00000100:00080000:3.0:1787757516.993806:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757516.993807:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993810:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757516.993817:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757516.993824:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:1.0:1787757516.993827:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659655808:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757516.993828:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757516.993834:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757516.993843:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757516.993845:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757516.993848:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757516.993852:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757516.993855:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787757516.993859:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757516.993862:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993865:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.993870:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757516.993871:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757516.993880:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757516.993883:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.993884:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993887:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757516.993902:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757516.993906:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993909:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.993915:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757516.993921:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 02000000:00000001:3.0:1787757516.993922:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757516.993927:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.993929:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757516.993932:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757516.993937:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757516.993940:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993948:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757516.993955:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659656064/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757516.993965:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757516.993968:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.993971:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757516.993975:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757516.993981:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757516.993983:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757516.993985:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757516.993989:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757516.993992:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10571 00000100:00080000:3.0:1787757516.994002:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757516.994003:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757516.994010:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757516.994015:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757516.994019:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757516.994022:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757516.994025:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757516.994028:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757516.994030:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757516.994032:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757516.994037:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757516.994042:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757516.994055:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994058:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757516.994059:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757516.994089:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757516.994094:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757516.994094:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.994105:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757516.994106:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:3.0:1787757516.994120:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757516.994126:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994129:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.994131:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994145:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305244865 02000000:00000001:3.0:1787757516.994146:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757516.994153:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.994157:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757516.994159:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757516.994161:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994166:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757516.994170:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994180:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305244865 00000100:00100000:3.0:1787757516.994233:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659656192/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757516.994235:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757516.994246:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757516.994256:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757516.994257:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757516.994264:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994266:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757516.994270:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757516.994281:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757516/real 1787757516] req@ffff8ffb45c33480 x1874588659655808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757516.994299:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757516.994303:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757516.994309:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757516.994313:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757516.994316:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994320:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757516.994323:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757516.994329:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659655808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757516.994340:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994345:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757516.994348:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994350:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757516.994352:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994355:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757516.994358:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757516.994363:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757516.994368:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757516.994371:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757516.994376:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659655808:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757516.994383:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994386:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757516.994388:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757516.994391:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757516.994397:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757516.994400:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757516.994403:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757516.994409:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757516.994411:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757516.994414:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757516.994416:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994419:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994430:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757516.994433:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994436:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757516.994438:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757516.994441:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757516.994443:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757516.994445:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.994448:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757516.994451:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757516.994454:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757516.994458:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655936:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757516.994463:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757516.994465:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757516.994467:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.994469:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757516.994473:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994480:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757516.994482:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757516.994489:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757516.994507:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757516.994513:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757516.994516:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757516.994518:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994520:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757516.994524:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994529:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305244174 00000100:00000001:1.0:1787757516.994534:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305244865 00000100:00000001:1.0:1787757516.994537:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305244174 00000100:00000001:1.0:1787757516.994540:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305244865 00000100:00000001:1.0:1787757516.994543:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757516.994545:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994547:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757516.994552:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757516/real 1787757516] req@ffff8ffb45c31180 x1874588659655936/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757516.994562:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757516.994564:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757516.994566:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757516.994568:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757516.994570:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994572:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757516.994574:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757516.994579:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659655936/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757516.994589:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994592:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757516.994595:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994596:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757516.994598:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994600:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757516.994621:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757516.994624:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757516.994628:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757516.994630:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757516.994634:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659655936:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757516.994638:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757516.994640:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757516.994642:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.994645:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757516.994647:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757516.994650:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757516.994653:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757516.994657:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757516.994659:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757516.994660:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.994662:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757516.994666:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994671:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757516.994673:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757516.994678:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757516.994692:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757516.994700:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757516.994704:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757516.994708:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994712:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757516.994717:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994723:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305244171 00000100:00000001:1.0:1787757516.994728:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305244865 00000100:00000001:1.0:1787757516.994733:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305244171 00000100:00000001:1.0:1787757516.994737:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305244865 00000100:00000001:1.0:1787757516.994741:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757516.994746:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994749:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757516.994758:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757516/real 1787757516] req@ffff8ffb45c33800 x1874588659656064/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757516.994776:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757516.994779:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757516.994784:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757516.994787:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757516.994790:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994794:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757516.994797:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757516.994804:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659656064/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757516.994814:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994817:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757516.994820:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994822:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757516.994823:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994826:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757516.994843:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757516.994846:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757516.994849:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757516.994852:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757516.994857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757516.994862:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757516.994864:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757516.994866:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.994868:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757516.994871:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757516.994873:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757516.994877:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757516.994881:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757516.994883:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757516.994886:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757516.994888:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757516.994892:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994899:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757516.994901:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757516.994906:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757516.994923:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757516.994928:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757516.994932:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757516.994934:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994936:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757516.994939:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994943:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305244171 00000100:00000001:1.0:1787757516.994947:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305244865 00000100:00000001:1.0:1787757516.994950:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305244171 00000100:00000001:1.0:1787757516.994953:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305244865 00000100:00000001:1.0:1787757516.994956:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757516.994958:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.994960:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757516.994965:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757516/real 1787757516] req@ffff8ffb45c31500 x1874588659656192/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757516.994983:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757516.994985:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757516.994988:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757516.994990:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757516.994991:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.994993:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757516.994995:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757516.994999:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659656192/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757516.995008:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.995011:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757516.995014:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.995016:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757516.995017:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.995019:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757516.995036:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757516.995039:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757516.995042:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757516.995045:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757516.995049:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757516.995053:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.995056:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757516.995058:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757516.995060:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757516.995063:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757516.995065:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757516.995066:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757516.995094:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757516.995096:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757516.995099:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757516.995101:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.995104:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757516.995105:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757516.995107:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757516.995109:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757516.995111:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757516.995113:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757516.995115:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757516.995116:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757516.995118:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757516.995119:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.995121:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757516.995123:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757516.995124:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757516.995127:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757516.995128:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757516.995130:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757516.995131:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757516.995133:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757516.995134:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757516.995136:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.995138:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757516.995142:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757516.995144:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757516.995146:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757516.995149:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757517.569312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757517.569324:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757517.569327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757517.569333:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757517.569338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757517.569343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757517.569346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757517.569350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757517.569353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757517.761416:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757517.761444:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757517.761453:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757517.761455:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757517.761525:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757517.761532:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757517.761536:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757518.593244:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757518.593249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757518.593251:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757518.593255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757518.593257:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757518.593260:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757518.593263:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757518.593265:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757518.593267:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757518.785344:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757518.785356:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757518.785367:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757518.785370:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757518.785405:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757518.785411:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757518.785417:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757519.617299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757519.617308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757519.617311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757519.617318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757519.617323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757519.617327:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757519.617330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757519.617333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757519.617337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757519.809355:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757519.809366:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757519.809379:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757519.809383:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757519.809489:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757519.809495:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757519.809500:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757520.641317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757520.641329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757520.641334:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757520.641343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757520.641351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757520.641357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757520.641363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757520.641367:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757520.641370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757520.833340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757520.833349:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757520.833358:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757520.833361:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757520.833496:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757520.833507:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757520.833516:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757521.665323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757521.665332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757521.665335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757521.665341:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757521.665346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757521.665351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757521.665354:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757521.665357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757521.665361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757521.857385:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757521.857395:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757521.857404:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757521.857408:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757521.857411:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757521.857414:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757521.857418:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757521.857420:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757521.857423:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757521.857426:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757521.857538:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757521.857547:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757521.857553:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757522.113369:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757522.113387:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757522.113391:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757522.113397:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757522.113404:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757522.113416:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10576 00000100:00080000:1.0:1787757522.113443:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757522.113448:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757522.113451:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757522.113457:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757522.113461:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757522.113466:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757522.113470:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757522.113473:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757522.113475:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757522.113482:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757522.113486:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757522.113489:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757522.113495:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757522.113499:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113502:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757522.113506:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757522.113524:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757522.113527:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113531:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757522.113536:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757522.113543:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113562:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757522.113576:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659656320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757522.113592:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757522.113598:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113631:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757522.113635:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757522.113640:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757522.113643:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757522.113645:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757522.113649:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757522.113652:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10576 00000100:00080000:1.0:1787757522.113656:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757522.113659:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757522.113660:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757522.113664:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757522.113665:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757522.113668:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757522.113672:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757522.113675:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757522.113677:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757522.113683:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757522.113686:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757522.113688:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757522.113696:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757522.113699:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113711:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757522.113718:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757522.113722:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113725:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757522.113731:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757522.113734:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113736:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757522.113738:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757522.113741:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.113747:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757522.113748:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.113765:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:1.0:1787757522.113769:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659656448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757522.113774:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757522.113786:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757522.113795:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113798:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757522.113811:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.113812:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757522.113821:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.113822:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757522.113834:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787757522.113834:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757522.113851:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757522.113855:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757522.113865:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757522.113866:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659656320:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757522.113879:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757522.113891:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757522.113892:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10576 00000100:00080000:1.0:1787757522.113903:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:2.0:1787757522.113905:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757522.113914:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787757522.113917:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.113924:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757522.113925:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:1.0:1787757522.113934:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757522.113936:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757522.113939:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757522.113942:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757522.113964:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757522.113968:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757522.113971:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757522.113973:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:1.0:1787757522.114001:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757522.114032:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757522.114033:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757522.114043:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757522.114049:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757522.114057:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757522.114062:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.114067:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757522.114072:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757522.114078:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.114084:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757522.114097:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000800:00000100:2.0:1787757522.114104:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:1.0:1787757522.114105:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.114114:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757522.114118:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757522.114120:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787757522.114126:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114131:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757522.114136:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114138:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757522.114146:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114148:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114164:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305249985 00000100:00100000:1.0:1787757522.114177:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659656576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757522.114231:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757522.114241:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305249986 00000100:00000001:2.0:1787757522.114256:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757522.114260:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757522.114263:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757522.114269:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.114272:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114276:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757522.114287:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757522.114300:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757522/real 1787757522] req@ffff8ffb4a536300 x1874588659656320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757522.114302:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757522.114325:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757522.114333:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757522.114340:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000400:00000001:2.0:1787757522.114349:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787757522.114350:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000400:00000001:2.0:1787757522.114358:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:1.0:1787757522.114369:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10576 00000100:00000001:2.0:1787757522.114388:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:1.0:1787757522.114393:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757522.114396:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757522.114403:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757522.114406:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757522.114409:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757522.114418:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757522.114419:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757522.114435:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:2.0:1787757522.114438:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659656320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:1.0:1787757522.114443:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757522.114461:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757522.114479:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757522.114481:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757522.114491:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757522.114492:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757522.114496:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757522.114505:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757522.114517:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757522.114519:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757522.114526:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757522.114530:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114533:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:1.0:1787757522.114536:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757522.114538:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:1.0:1787757522.114548:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787757522.114550:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:1.0:1787757522.114571:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757522.114578:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757522.114580:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757522.114587:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757522.114588:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000008:00000001:1.0:1787757522.114601:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.114607:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757522.114612:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659656320:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757522.114616:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757522.114620:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.114630:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757522.114635:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757522.114639:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114641:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.114649:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757522.114652:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114656:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757522.114664:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00100000:1.0:1787757522.114666:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659656704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757522.114674:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757522.114682:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:1.0:1787757522.114683:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000020:00000001:2.0:1787757522.114694:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:1.0:1787757522.114695:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757522.114705:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.114706:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757522.114717:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757522.114725:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757522.114728:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.114732:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.114741:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757522.114744:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114750:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757522.114752:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757522.114754:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757522.114756:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757522.114758:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.114762:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757522.114766:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757522.114769:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757522.114778:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656448:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757522.114784:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757522.114788:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757522.114790:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.114792:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757522.114796:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114805:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757522.114806:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787757522.114812:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757522.114834:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757522.114840:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757522.114843:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757522.114845:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114847:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757522.114855:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114860:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305249294 00000100:00000001:2.0:1787757522.114864:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305249985 00000100:00000001:2.0:1787757522.114867:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305249294 00000100:00000001:2.0:1787757522.114871:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305249986 00000100:00000001:2.0:1787757522.114873:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757522.114878:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.114881:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757522.114888:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757522/real 1787757522] req@ffff8ffb4a537800 x1874588659656448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757522.114899:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757522.114901:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757522.114904:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757522.114906:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757522.114910:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.114915:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757522.114918:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757522.114923:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659656448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757522.114932:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.114935:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757522.114938:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.114940:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757522.114942:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.114944:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757522.114960:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757522.114963:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757522.114967:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757522.114970:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757522.114974:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656448:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757522.115008:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757522.115012:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757522.115014:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.115018:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757522.115021:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757522.115024:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757522.115029:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757522.115033:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757522.115036:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757522.115038:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.115040:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757522.115047:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115056:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757522.115058:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787757522.115063:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757522.115084:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757522.115096:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757522.115102:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757522.115106:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115110:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757522.115123:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115128:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305249291 00000100:00000001:2.0:1787757522.115131:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305249985 00000100:00000001:2.0:1787757522.115134:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305249291 00000100:00000001:2.0:1787757522.115137:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305249986 00000100:00000001:2.0:1787757522.115139:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757522.115142:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115145:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757522.115151:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757522/real 1787757522] req@ffff8ffb4a537b80 x1874588659656576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757522.115169:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757522.115171:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757522.115215:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757522.115218:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757522.115220:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115223:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757522.115226:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757522.115234:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659656576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757522.115246:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115249:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757522.115252:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115254:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757522.115256:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115261:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757522.115290:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757522.115296:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757522.115303:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757522.115308:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757522.115314:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757522.115324:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757522.115328:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757522.115331:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.115336:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757522.115340:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757522.115345:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757522.115351:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757522.115358:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757522.115364:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757522.115368:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757522.115372:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757522.115377:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115388:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757522.115391:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757522.115396:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757522.115414:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757522.115421:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757522.115426:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757522.115428:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115431:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757522.115436:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115440:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305249292 00000100:00000001:2.0:1787757522.115444:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305249985 00000100:00000001:2.0:1787757522.115447:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305249292 00000100:00000001:2.0:1787757522.115450:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305249986 00000100:00000001:2.0:1787757522.115452:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757522.115455:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115457:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757522.115463:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757522/real 1787757522] req@ffff8ffb4a534700 x1874588659656704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757522.115472:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757522.115473:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757522.115476:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757522.115478:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757522.115480:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115482:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757522.115483:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757522.115487:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659656704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757522.115495:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115497:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757522.115500:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115502:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757522.115503:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115505:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757522.115521:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757522.115524:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757522.115528:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757522.115530:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757522.115533:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757522.115537:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757522.115540:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757522.115543:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757522.115546:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757522.115548:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757522.115549:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757522.115553:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757522.115555:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757522.115557:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757522.115559:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115561:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757522.115562:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757522.115564:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757522.115567:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757522.115568:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757522.115569:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757522.115572:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757522.115573:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757522.115574:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757522.115576:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115577:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757522.115579:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757522.115580:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757522.115582:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757522.115583:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757522.115585:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757522.115587:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757522.115590:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757522.115592:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757522.115594:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115598:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757522.115604:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757522.115607:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757522.115610:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757522.115613:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757522.689268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757522.689276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757522.689279:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757522.689285:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757522.689289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757522.689293:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757522.689297:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757522.689300:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757522.689303:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757522.881279:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757522.881297:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757522.881303:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757522.881305:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757522.881340:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757522.881343:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757522.881345:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757523.713262:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757523.713271:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757523.713274:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757523.713280:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757523.713284:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757523.713288:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757523.713292:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757523.713294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757523.713297:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757523.905267:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757523.905275:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757523.905278:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757523.905282:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757523.905288:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757523.905290:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757523.905293:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757523.905296:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757523.905299:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757523.905301:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757523.905353:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757523.905357:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757523.905359:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757524.737290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757524.737299:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757524.737302:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757524.737307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757524.737311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757524.737315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757524.737318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757524.737321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757524.737323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757524.929291:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757524.929299:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757524.929306:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757524.929309:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757524.929405:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757524.929411:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757524.929415:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757525.761397:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757525.761413:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757525.761419:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757525.761430:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757525.761437:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757525.761447:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757525.761452:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757525.761458:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757525.761463:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757525.953374:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757525.953416:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757525.953428:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757525.953431:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757525.953517:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757525.953526:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757525.953532:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757526.785311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757526.785320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757526.785323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757526.785329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757526.785334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757526.785339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757526.785343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757526.785346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757526.785349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757526.977312:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757526.977321:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757526.977329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757526.977331:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757526.977357:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757526.977361:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757526.977364:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757527.233328:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757527.233341:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757527.233345:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757527.233350:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757527.233356:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757527.233364:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10581 00000100:00080000:1.0:1787757527.233377:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757527.233382:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757527.233385:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757527.233390:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757527.233393:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757527.233398:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757527.233401:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757527.233404:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757527.233405:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757527.233411:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757527.233415:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757527.233418:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757527.233422:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757527.233426:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233428:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757527.233431:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757527.233445:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757527.233448:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233451:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757527.233455:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757527.233460:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233475:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757527.233487:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659656832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757527.233501:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757527.233507:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233529:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757527.233533:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757527.233541:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757527.233543:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757527.233545:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757527.233549:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757527.233552:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10581 00000100:00080000:1.0:1787757527.233556:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757527.233558:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757527.233560:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757527.233562:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757527.233564:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757527.233567:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757527.233570:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757527.233573:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757527.233574:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757527.233578:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757527.233581:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757527.233583:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757527.233589:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757527.233591:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233594:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757527.233599:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757527.233603:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233605:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757527.233610:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757527.233611:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233614:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757527.233616:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757527.233619:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233625:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757527.233632:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659656960/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757527.233652:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757527.233653:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757527.233657:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.233662:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757527.233666:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.233669:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:1.0:1787757527.233673:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757527.233676:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757527.233683:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757527.233684:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757527.233691:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757527.233695:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787757527.233696:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757527.233700:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:2.0:1787757527.233704:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757527.233708:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10581 02000000:00000001:2.0:1787757527.233713:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757527.233716:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757527.233723:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757527.233728:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787757527.233730:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659656832:192.168.201.118@tcp:250:kworker.0 00000020:00000001:1.0:1787757527.233732:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757527.233737:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757527.233743:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757527.233746:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:1.0:1787757527.233749:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757527.233755:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787757527.233756:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757527.233758:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757527.233763:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757527.233770:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787757527.233772:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757527.233776:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757527.233780:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757527.233784:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757527.233785:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757527.233796:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233799:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757527.233803:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757527.233806:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233809:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.233814:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787757527.233817:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757527.233821:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 02000000:00000001:1.0:1787757527.233822:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233830:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757527.233831:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787757527.233833:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757527.233840:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233848:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757527.233855:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659657088/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757527.233865:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757527.233868:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.233873:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757527.233876:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757527.233879:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757527.233882:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757527.233884:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757527.233928:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757527.233933:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10581 00000100:00000001:2.0:1787757527.233938:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757527.233939:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757527.233949:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757527.233951:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757527.233953:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305255105 00000020:00000001:1.0:1787757527.233955:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757527.233961:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757527.233964:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757527.233967:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757527.233970:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757527.233972:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757527.233973:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787757527.233980:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757527.233987:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305255105 00000100:00000001:1.0:1787757527.233995:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.233996:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787757527.234003:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757527.234007:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757527.234009:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757527.234015:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757527.234017:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.234023:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757527.234025:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787757527.234027:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757527.234035:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:1.0:1787757527.234036:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.234046:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234052:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:1.0:1787757527.234055:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757527.234058:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 02000000:00000001:1.0:1787757527.234060:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234067:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757527.234068:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757527.234074:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757527.234075:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787757527.234079:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234082:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757527.234088:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757527.234089:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.234092:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757527.234094:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757527.234103:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787757527.234108:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659657216/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757527.234112:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.234130:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:2.0:1787757527.234134:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656960:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757527.234136:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757527.234146:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234154:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757527.234162:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757527.234166:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757527.234171:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757527.234179:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234259:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757527.234262:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787757527.234270:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757527.234290:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234296:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305254414 00000100:00000001:2.0:1787757527.234300:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305255105 00000100:00000001:2.0:1787757527.234303:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305254414 00000100:00000001:2.0:1787757527.234305:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305255105 00000100:00000001:2.0:1787757527.234308:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757527.234311:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234313:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757527.234315:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757527.234317:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757527.234320:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757527.234322:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757527.234325:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757527.234331:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757527.234336:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757527.234339:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757527.234341:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757527.234343:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757527.234346:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234352:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757527.234354:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757527.234358:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757527.234366:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234370:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305254411 00000100:00000001:2.0:1787757527.234372:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305255105 00000100:00000001:2.0:1787757527.234375:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305254411 00000100:00000001:2.0:1787757527.234378:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305255105 00000100:00000001:2.0:1787757527.234380:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757527.234383:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234385:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234388:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757527.234393:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757527.234395:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787757527.234397:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757527.234399:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757527.234401:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757527.234402:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757527.234404:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757527.234406:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757527.234408:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757527.234410:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757527.234414:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757527.234418:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757527.234420:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757527.234421:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757527.234423:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757527.234426:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234431:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757527.234433:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787757527.234436:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757527.234445:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234449:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305254411 00000100:00000001:2.0:1787757527.234452:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305255105 00000100:00000001:2.0:1787757527.234454:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305254411 00000100:00000001:2.0:1787757527.234457:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305255105 00000100:00000001:2.0:1787757527.234460:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757527.234462:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234465:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234467:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757527.234469:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757527.234471:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787757527.234473:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757527.234474:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757527.234476:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757527.234479:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757527.809240:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757527.809247:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757527.809249:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757527.809253:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757527.809255:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757527.809258:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757527.809260:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757527.809262:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757527.809265:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757528.001346:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757528.001355:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757528.001366:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757528.001369:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757528.001408:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757528.001414:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757528.001420:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757528.833400:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757528.833412:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757528.833415:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757528.833422:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757528.833427:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757528.833433:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757528.833437:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757528.833441:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757528.833445:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757529.025306:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757529.025317:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757529.025327:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757529.025330:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757529.025379:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757529.025385:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757529.025389:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757529.857399:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757529.857411:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757529.857415:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757529.857422:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757529.857428:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757529.857434:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757529.857438:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757529.857442:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757529.857446:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757530.049326:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757530.049337:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757530.049347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757530.049350:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757530.049391:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757530.049398:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757530.049404:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787757530.305507:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787757530.305554:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787757530.305571:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757530.305585:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757530.305602:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757530.305605:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757530.305607:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757530.305617:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757530.305622:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757530.305624:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757530.305625:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757530.305627:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757530.305630:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757530.305635:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757530.305638:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757530.305639:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757530.305641:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757530.305645:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757530.305661:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757530.305668:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757530.305671:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757530.305672:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757530.305674:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757530.305681:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757530.305686:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757530.305709:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757527/real 1787757530] req@ffff8ffb4a537b80 x1874588659657216/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757530.305730:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757530.305733:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757530.305741:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757530.305745:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757530.305748:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.305752:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757530.305755:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757530.305762:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659657216/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757530.305777:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.305782:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757530.305786:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.305789:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757530.305791:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.305795:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757530.305815:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757530.305820:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757530.305828:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757530.305874:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757530.305883:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757530.305891:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757530.305898:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757527/real 1787757530] req@ffff8ffb4a535c00 x1874588659656960/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757530.305913:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757530.305915:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757530.305920:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757530.305923:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757530.305925:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.305928:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757530.305930:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757530.305936:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659656960/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757530.305948:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.305951:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757530.305955:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.305958:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757530.305960:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.305964:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757530.305982:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757530.305987:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757530.305993:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757530.305997:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757530.306002:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659656960:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757530.306008:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757530.306015:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757527/real 1787757530] req@ffff8ffb4a534700 x1874588659657088/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757530.306028:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757530.306030:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757530.306034:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757530.306036:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757530.306038:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306041:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757530.306043:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757530.306050:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659657088/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757530.306059:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306062:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757530.306065:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306068:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757530.306069:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306072:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757530.306097:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757530.306102:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757530.306107:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757530.306110:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757530.306115:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757530.306122:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757530.306128:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757527/real 1787757530] req@ffff8ffb4a534a80 x1874588659656832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757530.306140:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757530.306142:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757530.306147:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757530.306150:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757530.306152:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306154:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757530.306156:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757530.306161:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659656832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757530.306172:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306175:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757530.306179:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306182:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757530.306222:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306227:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757530.306230:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757530.306233:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757530.306239:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757530.306242:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757530.306248:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659656832:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757530.306279:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306283:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757530.306285:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757530.306289:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757530.306297:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757530.306301:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757530.306305:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757530.306311:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757530.306315:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757530.306319:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757530.306321:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306324:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757530.306327:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757530.306329:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757530.306333:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757530.306335:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757530.306336:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757530.306341:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757530.306343:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757530.306346:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757530.306348:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306350:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757530.306352:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757530.306354:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757530.306356:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757530.306358:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757530.306360:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757530.306363:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757530.306365:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757530.306368:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757530.306370:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306373:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757530.306375:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757530.306377:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757530.306379:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757530.306381:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757530.306382:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757530.306385:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757530.306387:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757530.306389:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757530.306392:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306395:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757530.306407:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757530.306411:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757530.306414:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757530.306416:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757530.881262:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757530.881269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757530.881272:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757530.881278:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757530.881282:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757530.881285:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757530.881287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757530.881288:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757530.881290:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757531.073313:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757531.073320:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757531.073324:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757531.073336:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757531.073352:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757531.073354:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757531.073356:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757531.905288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757531.905296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757531.905299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757531.905305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757531.905309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757531.905314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757531.905317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757531.905320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757531.905322:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757532.097323:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757532.097333:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757532.097340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757532.097345:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757532.097349:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757532.097352:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757532.097356:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757532.097360:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757532.097362:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757532.097364:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757532.097429:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757532.097435:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757532.097439:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757532.353309:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757532.353318:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757532.353321:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757532.353323:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757532.353327:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757532.353333:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10587 00000100:00080000:3.0:1787757532.353342:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757532.353344:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757532.353346:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757532.353349:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757532.353358:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757532.353360:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757532.353362:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757532.353363:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757532.353364:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757532.353368:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757532.353369:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757532.353371:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757532.353374:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757532.353376:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353378:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757532.353380:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757532.353387:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757532.353388:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353391:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757532.353393:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757532.353408:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353418:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757532.353425:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659657344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757532.353433:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757532.353437:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353451:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757532.353454:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757532.353456:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757532.353457:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757532.353458:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757532.353460:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757532.353462:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10587 00000100:00080000:3.0:1787757532.353464:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757532.353465:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757532.353466:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757532.353468:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757532.353469:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757532.353470:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757532.353472:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757532.353473:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757532.353474:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757532.353476:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757532.353478:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757532.353479:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757532.353483:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757532.353485:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353486:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757532.353491:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757532.353493:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353495:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757532.353498:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757532.353499:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353511:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757532.353511:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757532.353513:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757532.353516:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353518:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757532.353520:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353524:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:3.0:1787757532.353526:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659657472/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757532.353528:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757532.353532:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353537:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757532.353539:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353542:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757532.353543:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787757532.353544:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757532.353546:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:2.0:1787757532.353548:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757532.353549:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757532.353552:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787757532.353554:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353555:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757532.353562:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10587 00000100:00100000:2.0:1787757532.353563:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659657344:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757532.353566:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757532.353572:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757532.353574:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757532.353575:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757532.353590:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:2.0:1787757532.353591:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787757532.353592:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787757532.353597:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757532.353597:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757532.353605:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787757532.353605:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757532.353610:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757532.353612:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757532.353616:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757532.353619:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757532.353620:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757532.353621:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757532.353626:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757532.353631:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353633:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757532.353636:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757532.353641:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00000001:3.0:1787757532.353641:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353645:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757532.353646:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353654:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757532.353654:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757532.353658:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353662:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757532.353663:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757532.353666:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353672:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757532.353678:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659657600/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757532.353686:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757532.353688:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353691:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757532.353693:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757532.353696:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757532.353698:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757532.353702:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787757532.353703:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757532.353707:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757532.353713:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10587 00000100:00000001:2.0:1787757532.353717:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757532.353718:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757532.353723:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757532.353725:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757532.353726:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757532.353730:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757532.353731:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757532.353736:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:3.0:1787757532.353737:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757532.353741:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757532.353744:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757532.353745:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353748:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757532.353751:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757532.353753:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305260225 00000100:00080000:3.0:1787757532.353757:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757532.353762:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353766:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757532.353767:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757532.353772:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757532.353776:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305260225 02000000:00000001:3.0:1787757532.353777:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353782:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757532.353784:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00100000:3.0:1787757532.353785:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757532.353791:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757532.353793:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353813:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757532.353818:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353821:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757532.353827:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757532/real 1787757532] req@ffff8ffb45c32a00 x1874588659657344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757532.353837:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:2.0:1787757532.353840:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787757532.353841:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757532.353844:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757532.353846:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757532.353851:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787757532.353854:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757532.353856:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787757532.353859:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353861:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.353865:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757532.353867:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353868:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757532.353879:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659657728/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757532.353881:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659657344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757532.353907:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757532.353909:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757532.353912:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353915:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757532.353917:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353922:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.353926:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757532.353928:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.353931:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757532.353933:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757532.353936:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757532.353942:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757532.353945:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757532.353951:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659657344:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757532.353957:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.353960:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757532.353962:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757532.353964:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757532.353968:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757532.353971:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757532.353973:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757532.353976:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757532.353978:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757532.353981:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757532.353982:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.353985:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.353992:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757532.353994:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.353996:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757532.353998:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757532.354000:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757532.354001:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757532.354003:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757532.354006:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757532.354007:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757532.354009:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757532.354013:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657472:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757532.354016:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757532.354018:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757532.354020:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757532.354021:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757532.354025:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354031:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757532.354033:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787757532.354037:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757532.354050:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757532.354055:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757532.354057:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757532.354059:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354061:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757532.354064:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354068:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305259533 00000100:00000001:2.0:1787757532.354071:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305260225 00000100:00000001:2.0:1787757532.354073:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305259533 00000100:00000001:2.0:1787757532.354075:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305260225 00000100:00000001:2.0:1787757532.354077:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757532.354079:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354080:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757532.354085:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757532/real 1787757532] req@ffff8ffb45c30700 x1874588659657472/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757532.354093:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757532.354095:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757532.354097:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757532.354098:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757532.354099:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354102:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757532.354103:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757532.354106:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659657472/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757532.354114:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354116:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757532.354119:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354120:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757532.354121:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354123:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757532.354137:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757532.354140:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757532.354143:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757532.354145:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757532.354149:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657472:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757532.354153:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757532.354154:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757532.354155:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757532.354157:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757532.354159:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757532.354162:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757532.354165:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757532.354168:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757532.354170:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757532.354172:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757532.354174:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757532.354177:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354216:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757532.354218:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787757532.354223:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757532.354238:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757532.354245:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757532.354247:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757532.354249:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354251:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757532.354254:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354258:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305259531 00000100:00000001:2.0:1787757532.354261:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305260225 00000100:00000001:2.0:1787757532.354263:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305259531 00000100:00000001:2.0:1787757532.354266:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305260225 00000100:00000001:2.0:1787757532.354268:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757532.354270:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354272:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757532.354277:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757532/real 1787757532] req@ffff8ffb45c30000 x1874588659657600/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757532.354286:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757532.354287:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757532.354291:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757532.354293:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757532.354294:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354296:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757532.354298:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757532.354302:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659657600/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757532.354310:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354312:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757532.354314:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354316:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757532.354317:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354319:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757532.354331:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757532.354334:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757532.354338:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757532.354340:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757532.354344:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757532.354348:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757532.354349:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757532.354350:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757532.354352:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757532.354354:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757532.354356:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757532.354359:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757532.354362:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757532.354364:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757532.354365:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757532.354368:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757532.354371:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354378:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757532.354379:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:2.0:1787757532.354383:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757532.354396:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757532.354401:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757532.354404:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757532.354405:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354407:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757532.354410:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354414:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305259531 00000100:00000001:2.0:1787757532.354417:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305260225 00000100:00000001:2.0:1787757532.354420:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305259531 00000100:00000001:2.0:1787757532.354422:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305260225 00000100:00000001:2.0:1787757532.354424:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757532.354427:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354429:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757532.354434:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757532/real 1787757532] req@ffff8ffb45c30e00 x1874588659657728/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757532.354442:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757532.354443:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757532.354446:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757532.354448:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757532.354450:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354452:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757532.354453:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757532.354458:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659657728/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757532.354465:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354468:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757532.354471:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354473:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757532.354474:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354476:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757532.354490:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757532.354492:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757532.354496:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757532.354498:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757532.354502:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757532.354506:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354509:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757532.354510:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757532.354512:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757532.354514:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757532.354516:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757532.354518:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757532.354521:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757532.354522:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757532.354525:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757532.354527:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354529:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757532.354530:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757532.354532:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757532.354534:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757532.354535:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757532.354537:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757532.354539:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757532.354540:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757532.354541:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757532.354543:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354545:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757532.354546:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757532.354547:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757532.354550:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757532.354551:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757532.354552:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757532.354554:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757532.354556:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757532.354557:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757532.354558:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354560:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757532.354564:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757532.354566:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757532.354568:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757532.354570:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757532.929266:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757532.929272:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757532.929274:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757532.929279:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757532.929282:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757532.929285:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757532.929287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757532.929289:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757532.929291:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757533.121321:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757533.121331:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757533.121349:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757533.121352:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757533.121433:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757533.121439:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757533.121445:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757533.953360:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757533.953369:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757533.953372:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757533.953389:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757533.953393:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757533.953398:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757533.953401:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757533.953404:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757533.953407:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757534.145338:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757534.145351:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757534.145356:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757534.145361:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757534.145369:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757534.145374:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757534.145379:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757534.145383:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757534.145391:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757534.145394:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757534.145500:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757534.145507:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757534.145512:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757534.977415:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757534.977428:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757534.977431:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757534.977456:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757534.977461:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757534.977466:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757534.977470:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757534.977473:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757534.977476:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757535.169361:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757535.169371:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757535.169380:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757535.169381:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757535.169430:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757535.169434:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757535.169439:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757536.001268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757536.001276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757536.001278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757536.001283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757536.001287:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757536.001292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757536.001295:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757536.001299:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757536.001302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757536.193320:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757536.193340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757536.193349:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757536.193351:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757536.193453:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757536.193460:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757536.193466:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757537.025342:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757537.025353:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757537.025356:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757537.025363:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757537.025369:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757537.025376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757537.025381:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757537.025404:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757537.025410:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757537.217305:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757537.217314:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757537.217321:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757537.217322:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757537.217475:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757537.217487:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757537.217495:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757537.473339:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757537.473351:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757537.473353:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757537.473358:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757537.473362:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757537.473379:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10592 00000100:00080000:2.0:1787757537.473393:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757537.473397:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757537.473399:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757537.473404:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757537.473406:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757537.473410:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757537.473412:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757537.473414:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757537.473414:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757537.473419:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757537.473420:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757537.473423:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757537.473427:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757537.473430:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473432:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757537.473434:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757537.473450:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757537.473452:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473456:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757537.473459:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757537.473464:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473478:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757537.473490:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659657856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757537.473500:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757537.473504:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473525:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757537.473528:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757537.473531:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757537.473532:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757537.473534:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757537.473536:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757537.473537:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10592 00000100:00080000:2.0:1787757537.473539:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757537.473541:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757537.473543:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757537.473545:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757537.473545:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757537.473547:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757537.473549:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757537.473550:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757537.473551:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757537.473553:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757537.473554:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757537.473555:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757537.473561:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757537.473562:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473564:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757537.473568:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757537.473570:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473572:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757537.473575:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757537.473576:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473578:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757537.473579:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757537.473581:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473586:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757537.473591:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659657984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757537.473598:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757537.473600:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473602:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757537.473606:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757537.473608:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757537.473610:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757537.473611:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757537.473615:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757537.473616:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787757537.473619:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10592 00000100:00080000:2.0:1787757537.473627:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757537.473628:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757537.473636:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757537.473638:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757537.473641:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757537.473645:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757537.473647:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757537.473655:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757537.473660:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757537.473663:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:2.0:1787757537.473664:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787757537.473673:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787757537.473673:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757537.473685:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757537.473688:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757537.473694:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757537.473697:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757537.473703:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757537.473704:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757537.473725:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757537.473736:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473740:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757537.473743:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757537.473746:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473750:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757537.473754:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659657856:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757537.473758:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757537.473762:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473769:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757537.473773:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757537.473778:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757537.473778:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473789:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757537.473790:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757537.473796:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757537.473802:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659658112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757537.473803:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757537.473816:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757537.473819:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473821:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473843:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757537.473850:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757537.473853:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757537.473855:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757537.473857:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757537.473859:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757537.473862:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10592 00000100:00000001:1.0:1787757537.473864:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787757537.473867:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757537.473874:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:2.0:1787757537.473874:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757537.473878:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757537.473884:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757537.473889:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757537.473895:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787757537.473896:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757537.473902:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757537.473904:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757537.473904:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757537.473907:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757537.473909:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757537.473910:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757537.473912:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757537.473914:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473915:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757537.473917:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757537.473919:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473921:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757537.473925:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757537.473927:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473928:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757537.473930:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757537.473931:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473935:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757537.473938:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659658240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757537.473944:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757537.473964:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757537.473968:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757537.473988:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757537.474013:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757537.474023:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757537.474027:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474031:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757537.474042:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474050:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305265345 00000100:00000001:1.0:1787757537.474069:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757537.474079:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305265345 00000100:00000001:1.0:1787757537.474085:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757537.474102:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757537.474107:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474113:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757537.474126:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757537/real 1787757537] req@ffff8ffb47a3a300 x1874588659657856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757537.474150:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757537.474154:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757537.474161:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757537.474166:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757537.474170:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474175:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757537.474226:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757537.474235:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659657856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757537.474245:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474250:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757537.474254:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474256:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757537.474258:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474261:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757537.474264:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757537.474268:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757537.474274:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757537.474277:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757537.474282:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659657856:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757537.474288:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757537.474291:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757537.474293:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757537.474297:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757537.474301:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757537.474304:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757537.474308:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657984:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757537.474312:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757537.474315:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757537.474318:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757537.474320:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757537.474325:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474355:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757537.474357:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:1.0:1787757537.474364:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757537.474385:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757537.474392:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757537.474395:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757537.474397:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474400:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757537.474403:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474408:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305264654 00000100:00000001:1.0:1787757537.474412:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305265345 00000100:00000001:1.0:1787757537.474416:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305264654 00000100:00000001:1.0:1787757537.474419:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305265345 00000100:00000001:1.0:1787757537.474421:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757537.474424:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474426:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757537.474432:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757537/real 1787757537] req@ffff8ffb47a38700 x1874588659657984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757537.474442:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757537.474443:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757537.474446:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757537.474448:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757537.474451:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474453:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757537.474455:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757537.474460:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659657984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757537.474468:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474471:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757537.474474:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474476:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757537.474477:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474479:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757537.474493:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757537.474497:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757537.474500:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757537.474503:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757537.474507:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659657984:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757537.474512:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474515:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757537.474517:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757537.474520:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757537.474524:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757537.474527:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757537.474530:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757537.474535:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757537.474537:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757537.474542:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757537.474544:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474546:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757537.474548:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757537.474549:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757537.474551:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757537.474553:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757537.474554:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757537.474556:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757537.474557:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757537.474561:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757537.474563:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474566:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474574:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757537.474577:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474580:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757537.474583:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757537.474585:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757537.474587:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757537.474589:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757537.474592:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757537.474594:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757537.474597:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757537.474601:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757537.474605:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757537.474607:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757537.474609:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757537.474611:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757537.474614:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474620:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757537.474622:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787757537.474627:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757537.474641:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757537.474646:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757537.474650:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757537.474653:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474656:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757537.474662:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474668:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305264651 00000100:00000001:1.0:1787757537.474673:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305265345 00000100:00000001:1.0:1787757537.474677:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305264651 00000100:00000001:1.0:1787757537.474682:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305265345 00000100:00000001:1.0:1787757537.474686:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757537.474690:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474694:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757537.474703:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757537/real 1787757537] req@ffff8ffb47a39180 x1874588659658112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757537.474742:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757537.474745:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757537.474751:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757537.474755:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757537.474758:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474762:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757537.474765:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757537.474774:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659658112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757537.474790:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474794:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757537.474797:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474799:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757537.474801:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474804:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757537.474816:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757537.474819:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757537.474824:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757537.474826:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757537.474831:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757537.474836:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757537.474838:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757537.474840:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757537.474844:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757537.474846:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757537.474849:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757537.474852:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757537.474857:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757537.474860:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757537.474863:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757537.474865:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757537.474869:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474877:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757537.474879:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787757537.474884:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757537.474901:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757537.474908:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757537.474911:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757537.474912:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474915:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757537.474918:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474923:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305264651 00000100:00000001:1.0:1787757537.474926:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305265345 00000100:00000001:1.0:1787757537.474929:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305264651 00000100:00000001:1.0:1787757537.474932:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305265345 00000100:00000001:1.0:1787757537.474934:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757537.474937:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.474939:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757537.474944:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757537/real 1787757537] req@ffff8ffb47a3b480 x1874588659658240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757537.474953:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757537.474954:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757537.474966:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757537.474968:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757537.474970:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474972:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757537.474974:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757537.474979:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659658240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757537.474987:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474990:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757537.474993:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474996:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757537.474997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.474999:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757537.475013:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757537.475017:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757537.475020:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757537.475023:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757537.475027:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757537.475031:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.475034:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757537.475035:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757537.475037:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757537.475040:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757537.475042:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757537.475043:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757537.475046:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757537.475047:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757537.475052:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757537.475054:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.475056:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757537.475057:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757537.475059:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757537.475061:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757537.475062:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757537.475063:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757537.475066:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757537.475067:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757537.475070:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757537.475072:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.475074:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757537.475078:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757537.475080:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757537.475083:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757537.475085:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757538.049321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757538.049329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757538.049332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757538.049339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757538.049343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757538.049348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757538.049351:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757538.049353:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757538.049357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757538.241373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757538.241387:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757538.241400:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757538.241406:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757538.241440:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757538.241449:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757538.241457:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757539.073276:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757539.073289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757539.073294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757539.073301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757539.073306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757539.073312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757539.073315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757539.073318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757539.073322:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757539.265326:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757539.265337:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757539.265347:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757539.265350:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757539.265385:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757539.265391:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757539.265395:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757540.097327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757540.097344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757540.097348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757540.097360:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757540.097368:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757540.097376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757540.097383:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757540.097389:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757540.097395:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757540.289340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757540.289351:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757540.289361:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757540.289365:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757540.289459:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757540.289466:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757540.289471:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757541.121407:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757541.121418:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757541.121422:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757541.121430:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757541.121435:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757541.121442:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757541.121446:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757541.121450:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757541.121454:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757541.313355:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757541.313367:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757541.313377:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757541.313384:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757541.313388:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757541.313394:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757541.313399:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757541.313402:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757541.313405:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757541.313410:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757541.313498:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757541.313507:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757541.313513:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757542.145322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757542.145335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757542.145338:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757542.145346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757542.145351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757542.145357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757542.145361:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757542.145365:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757542.145368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757542.337346:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757542.337356:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757542.337366:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757542.337368:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757542.337526:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757542.337532:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757542.337537:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757542.593302:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757542.593317:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757542.593321:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757542.593326:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757542.593333:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757542.593344:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10597 00000100:00080000:2.0:1787757542.593361:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757542.593366:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757542.593369:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757542.593376:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757542.593380:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757542.593384:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757542.593388:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757542.593391:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757542.593393:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757542.593400:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757542.593403:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757542.593406:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757542.593411:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757542.593415:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.593419:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757542.593422:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757542.593439:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757542.593442:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.593446:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757542.593450:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757542.593457:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.593475:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757542.593490:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659658368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757542.593506:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757542.593512:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.593536:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757542.593541:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757542.593546:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757542.593548:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757542.593550:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757542.593554:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757542.593557:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10597 00000100:00080000:2.0:1787757542.593561:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757542.593564:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757542.593566:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757542.593569:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757542.593571:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757542.593574:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757542.593578:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757542.593581:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757542.593583:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757542.593588:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757542.593593:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757542.593597:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757542.593608:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757542.593614:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757542.593638:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.593644:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757542.593647:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757542.593651:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757542.593657:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:2.0:1787757542.593660:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757542.593661:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.593673:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:2.0:1787757542.593675:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.593679:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757542.593684:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.593686:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757542.593700:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:1.0:1787757542.593701:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659658368:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757542.593708:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.593712:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757542.593719:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757542.593722:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757542.593726:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757542.593728:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757542.593736:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.593737:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757542.593749:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.593753:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757542.593766:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659658496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757542.593779:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757542.593779:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757542.593786:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:2.0:1787757542.593788:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.593797:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757542.593799:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757542.593807:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757542.593812:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757542.593814:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757542.593817:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757542.593821:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757542.593825:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10597 00000100:00080000:2.0:1787757542.593831:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757542.593834:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757542.593837:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757542.593841:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757542.593844:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757542.593847:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757542.593850:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757542.593854:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757542.593857:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:1.0:1787757542.593866:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787757542.593874:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757542.593882:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757542.593887:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757542.593888:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787757542.593895:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757542.593897:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757542.593903:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757542.593906:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.593908:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757542.593917:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757542.593921:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:2.0:1787757542.593924:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757542.593930:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305270465 00000008:00000001:2.0:1787757542.593937:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.593944:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.593950:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757542.593954:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757542.593961:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305270465 02000000:00000001:2.0:1787757542.593963:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.593973:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757542.593977:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757542.593978:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757542.593987:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.593999:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757542.594003:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594006:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594015:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757542.594022:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659658624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757542.594027:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757542/real 1787757542] req@ffff8ffb47a38000 x1874588659658368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757542.594061:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757542.594069:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757542.594070:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757542.594075:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757542.594077:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594085:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:2.0:1787757542.594085:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757542.594094:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:2.0:1787757542.594095:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757542.594102:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757542.594103:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757542.594109:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757542.594111:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757542.594118:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757542.594119:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757542.594132:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659658368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757542.594140:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10597 00000100:00000001:1.0:1787757542.594143:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757542.594153:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757542.594154:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757542.594163:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757542.594164:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757542.594169:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757542.594170:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757542.594175:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757542.594178:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757542.594220:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757542.594225:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:2.0:1787757542.594227:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00080000:1.0:1787757542.594233:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000020:00000001:2.0:1787757542.594234:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757542.594242:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757542.594247:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757542.594249:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757542.594257:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1787757542.594258:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757542.594270:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594274:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:2.0:1787757542.594276:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757542.594283:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757542.594291:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757542.594293:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659658368:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757542.594297:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757542.594303:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757542.594310:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:2.0:1787757542.594311:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.594322:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594324:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787757542.594329:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757542.594332:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757542.594334:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757542.594340:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757542.594342:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757542.594343:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757542.594350:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.594357:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:2.0:1787757542.594359:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757542.594367:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00100000:2.0:1787757542.594371:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659658752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000001:1.0:1787757542.594373:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757542.594384:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:2.0:1787757542.594387:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757542.594389:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757542.594395:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594399:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757542.594402:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594405:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594412:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594426:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757542.594429:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594433:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757542.594436:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757542.594438:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757542.594441:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757542.594443:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.594447:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757542.594450:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757542.594453:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757542.594459:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658496:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757542.594464:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757542.594467:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757542.594470:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.594472:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757542.594476:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594487:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757542.594489:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:1.0:1787757542.594495:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757542.594515:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757542.594522:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757542.594526:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757542.594528:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594530:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757542.594534:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594540:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305269774 00000100:00000001:1.0:1787757542.594544:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305270465 00000100:00000001:1.0:1787757542.594547:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305269774 00000100:00000001:1.0:1787757542.594550:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305270465 00000100:00000001:1.0:1787757542.594553:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757542.594555:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594558:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757542.594563:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757542/real 1787757542] req@ffff8ffb47a38380 x1874588659658496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757542.594574:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757542.594575:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757542.594578:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757542.594580:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757542.594583:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594585:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757542.594588:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757542.594593:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659658496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757542.594603:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594606:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757542.594609:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594610:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757542.594612:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594614:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757542.594667:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757542.594671:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757542.594675:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757542.594678:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757542.594682:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658496:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757542.594687:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757542.594689:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757542.594692:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.594695:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757542.594697:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757542.594700:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757542.594703:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757542.594708:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757542.594710:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757542.594712:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.594715:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757542.594719:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594727:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757542.594729:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757542.594735:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757542.594751:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757542.594758:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757542.594761:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757542.594763:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594766:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757542.594769:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594774:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305269771 00000100:00000001:1.0:1787757542.594777:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305270465 00000100:00000001:1.0:1787757542.594780:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305269771 00000100:00000001:1.0:1787757542.594783:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305270465 00000100:00000001:1.0:1787757542.594787:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757542.594789:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594791:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757542.594796:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757542/real 1787757542] req@ffff8ffb47a38a80 x1874588659658624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757542.594806:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757542.594807:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757542.594810:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757542.594812:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757542.594814:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594817:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757542.594818:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757542.594823:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659658624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757542.594831:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594834:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757542.594837:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594838:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757542.594840:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594842:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757542.594860:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757542.594863:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757542.594866:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757542.594870:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757542.594874:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757542.594879:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757542.594881:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757542.594883:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.594885:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757542.594887:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757542.594889:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757542.594893:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757542.594896:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757542.594898:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757542.594900:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757542.594902:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757542.594905:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594910:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757542.594912:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757542.594917:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757542.594928:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757542.594933:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757542.594936:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757542.594938:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594940:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757542.594943:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594947:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305269771 00000100:00000001:1.0:1787757542.594951:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305270465 00000100:00000001:1.0:1787757542.594954:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305269771 00000100:00000001:1.0:1787757542.594957:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305270465 00000100:00000001:1.0:1787757542.594959:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757542.594962:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.594964:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757542.594968:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757542/real 1787757542] req@ffff8ffb47a3a680 x1874588659658752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757542.594987:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757542.594989:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757542.594992:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757542.594994:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757542.594995:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.594997:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757542.594999:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757542.595003:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659658752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757542.595011:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.595014:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757542.595016:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.595018:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757542.595020:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.595023:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757542.595041:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757542.595045:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757542.595048:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757542.595052:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757542.595056:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659658752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757542.595060:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.595062:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757542.595064:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757542.595066:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757542.595069:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757542.595071:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757542.595072:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757542.595075:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757542.595076:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757542.595079:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757542.595080:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.595083:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757542.595085:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757542.595087:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757542.595089:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757542.595090:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757542.595092:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757542.595094:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757542.595095:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757542.595097:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757542.595098:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.595100:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757542.595102:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757542.595103:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757542.595106:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757542.595108:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757542.595109:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757542.595110:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757542.595112:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757542.595115:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757542.595117:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.595119:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757542.595123:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757542.595125:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757542.595127:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757542.595130:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757543.169322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757543.169337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757543.169342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757543.169352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757543.169358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757543.169364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757543.169368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757543.169372:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757543.169377:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757543.361417:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757543.361442:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757543.361452:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757543.361454:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757543.361484:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757543.361488:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757543.361492:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757544.193314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757544.193327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757544.193331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757544.193340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757544.193347:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757544.193353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757544.193358:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757544.193364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757544.193369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757544.385303:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757544.385318:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757544.385324:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757544.385330:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757544.385339:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757544.385343:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757544.385348:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757544.385354:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757544.385362:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757544.385364:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757544.385484:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757544.385494:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757544.385501:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757545.217326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757545.217338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757545.217342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757545.217351:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757545.217357:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757545.217364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757545.217369:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757545.217374:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757545.217379:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757545.409338:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757545.409350:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757545.409360:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757545.409362:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757545.409436:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757545.409447:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757545.409452:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757546.241382:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757546.241394:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757546.241397:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757546.241406:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757546.241411:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757546.241416:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757546.241420:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757546.241426:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757546.241429:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757546.433366:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757546.433377:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757546.433387:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757546.433389:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757546.433480:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757546.433486:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757546.433492:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757547.265293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757547.265303:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757547.265306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757547.265312:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757547.265317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757547.265321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757547.265324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757547.265327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757547.265331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757547.457327:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757547.457336:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757547.457345:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757547.457347:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757547.457389:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757547.457394:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757547.457400:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757547.713337:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757547.713353:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757547.713357:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757547.713364:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757547.713371:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757547.713383:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10602 00000100:00080000:2.0:1787757547.713400:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757547.713405:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757547.713408:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757547.713427:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757547.713432:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757547.713437:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757547.713441:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757547.713445:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757547.713447:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757547.713453:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757547.713457:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757547.713461:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757547.713466:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757547.713471:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.713474:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757547.713478:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757547.713497:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757547.713500:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.713505:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757547.713509:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757547.713517:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.713596:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757547.713620:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659658880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757547.713642:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757547.713651:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.713680:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757547.713685:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757547.713691:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757547.713693:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757547.713695:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757547.713700:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757547.713704:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10602 00000100:00080000:2.0:1787757547.713710:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757547.713713:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757547.713716:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757547.713721:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757547.713723:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757547.713727:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757547.713732:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757547.713736:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757547.713738:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757547.713742:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757547.713746:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757547.713748:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757547.713757:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757547.713757:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757547.713770:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.713774:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757547.713775:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757547.713788:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:2.0:1787757547.713790:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757547.713794:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:2.0:1787757547.713801:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.713803:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.713811:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.713816:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757547.713820:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757547.713827:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757547.713833:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757547.713837:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.713844:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:1.0:1787757547.713847:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659658880:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757547.713849:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757547.713859:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.713867:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757547.713876:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.713880:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757547.713887:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757547.713891:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659659008/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757547.713901:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757547.713914:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757547.713918:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.713922:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.713925:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757547.713938:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757547.713943:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757547.713946:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757547.713948:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757547.713952:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757547.713957:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10602 00000100:00000001:1.0:1787757547.713961:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787757547.713967:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757547.713970:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:2.0:1787757547.713977:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757547.713983:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757547.713986:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757547.713996:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757547.713999:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757547.714004:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757547.714009:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757547.714015:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757547.714017:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757547.714027:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757547.714034:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757547.714039:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000800:00000100:1.0:1787757547.714042:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:2.0:1787757547.714050:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757547.714060:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714061:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757547.714068:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757547.714075:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00100000:2.0:1787757547.714075:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757547.714083:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714090:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00000001:2.0:1787757547.714091:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714098:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.714101:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714106:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305275585 02000000:00000001:2.0:1787757547.714120:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757547.714122:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757547.714130:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714136:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305275585 00000100:00000001:2.0:1787757547.714140:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757547.714145:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757547.714148:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757547.714152:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757547.714158:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757547.714159:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714163:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757547.714179:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757547/real 1787757547] req@ffff8ffb47a39880 x1874588659658880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757547.714237:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757547.714259:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659659136/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757547.714310:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757547.714314:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757547.714323:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757547.714325:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757547.714328:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:2.0:1787757547.714335:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757547.714339:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:2.0:1787757547.714342:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757547.714347:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757547.714347:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757547.714357:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757547.714361:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757547.714364:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757547.714377:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:2.0:1787757547.714378:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10602 00000100:00000001:1.0:1787757547.714393:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757547.714397:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757547.714409:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757547.714413:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757547.714420:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757547.714423:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:1.0:1787757547.714424:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659658880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787757547.714427:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757547.714438:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757547.714452:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757547.714460:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757547.714482:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757547.714484:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757547.714501:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714502:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:2.0:1787757547.714518:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757547.714520:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.714554:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:2.0:1787757547.714555:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757547.714565:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757547.714567:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714573:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757547.714579:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757547.714579:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757547.714591:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757547.714592:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757547.714606:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00000001:2.0:1787757547.714608:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714613:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757547.714621:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757547.714624:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659658880:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757547.714637:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757547.714639:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757547.714642:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757547.714651:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757547.714651:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.714660:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:2.0:1787757547.714669:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757547.714672:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:2.0:1787757547.714673:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000020:00000001:1.0:1787757547.714682:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:2.0:1787757547.714684:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757547.714689:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757547.714702:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:2.0:1787757547.714704:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.714710:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757547.714719:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757547.714727:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757547.714729:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659659264/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757547.714743:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757547.714764:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757547.714765:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757547.714770:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757547.714771:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714788:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757547.714789:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714793:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757547.714798:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757547.714802:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757547.714804:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.714809:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757547.714813:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757547.714817:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757547.714823:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659008:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757547.714829:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757547.714834:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757547.714836:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.714839:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757547.714845:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714854:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757547.714856:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:1.0:1787757547.714862:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757547.714890:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757547.714900:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757547.714904:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757547.714907:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714910:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757547.714916:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714924:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305274894 00000100:00000001:1.0:1787757547.714931:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305275585 00000100:00000001:1.0:1787757547.714936:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305274894 00000100:00000001:1.0:1787757547.714940:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305275585 00000100:00000001:1.0:1787757547.714944:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757547.714949:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.714952:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757547.714963:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757547/real 1787757547] req@ffff8ffb47a39500 x1874588659659008/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757547.714982:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757547.714985:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757547.714990:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757547.714992:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757547.714994:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.714996:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757547.714998:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757547.715004:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659659008/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757547.715014:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715017:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757547.715020:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715022:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757547.715024:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715026:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757547.715043:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757547.715047:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757547.715052:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757547.715055:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757547.715059:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659008:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757547.715064:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757547.715067:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757547.715069:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.715072:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757547.715074:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757547.715077:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757547.715081:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757547.715085:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757547.715088:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757547.715090:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.715093:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757547.715097:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715105:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757547.715106:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757547.715112:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757547.715129:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757547.715135:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757547.715138:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757547.715140:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715142:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757547.715145:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715150:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305274891 00000100:00000001:1.0:1787757547.715153:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305275585 00000100:00000001:1.0:1787757547.715156:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305274891 00000100:00000001:1.0:1787757547.715159:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305275585 00000100:00000001:1.0:1787757547.715162:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757547.715164:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715167:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757547.715172:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757547/real 1787757547] req@ffff8ffb47a3a680 x1874588659659136/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757547.715182:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757547.715218:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757547.715222:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757547.715225:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757547.715227:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715229:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757547.715231:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757547.715236:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659659136/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757547.715245:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715248:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757547.715251:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715253:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757547.715255:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715257:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757547.715272:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757547.715275:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757547.715278:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757547.715281:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757547.715285:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757547.715289:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757547.715292:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757547.715294:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.715297:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757547.715299:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757547.715301:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757547.715305:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757547.715309:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757547.715312:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757547.715315:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757547.715317:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757547.715321:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715327:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757547.715329:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757547.715334:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757547.715349:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757547.715358:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757547.715362:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757547.715365:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715368:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757547.715373:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715379:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305274892 00000100:00000001:1.0:1787757547.715384:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305275585 00000100:00000001:1.0:1787757547.715389:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305274892 00000100:00000001:1.0:1787757547.715393:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305275585 00000100:00000001:1.0:1787757547.715397:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757547.715401:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715405:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757547.715414:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757547/real 1787757547] req@ffff8ffb47a38a80 x1874588659659264/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757547.715433:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757547.715435:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757547.715441:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757547.715444:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757547.715447:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715450:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757547.715453:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757547.715459:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659659264/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757547.715468:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715471:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757547.715475:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715477:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757547.715479:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715481:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757547.715496:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757547.715500:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757547.715504:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757547.715507:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757547.715511:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757547.715515:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715517:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757547.715519:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757547.715521:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757547.715550:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757547.715553:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757547.715555:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757547.715559:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757547.715560:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757547.715563:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757547.715565:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715568:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757547.715569:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757547.715571:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757547.715574:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757547.715576:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757547.715577:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757547.715579:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757547.715581:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757547.715584:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757547.715586:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715587:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757547.715589:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757547.715590:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757547.715592:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757547.715593:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757547.715594:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757547.715596:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757547.715598:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757547.715601:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757547.715603:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715605:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757547.715611:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757547.715613:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757547.715616:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757547.715618:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757548.289350:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757548.289363:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757548.289367:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757548.289375:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757548.289380:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757548.289386:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757548.289390:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757548.289394:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757548.289398:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757548.481363:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757548.481380:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757548.481395:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757548.481399:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757548.481478:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757548.481484:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757548.481490:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757549.313302:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757549.313314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757549.313318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757549.313326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757549.313332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757549.313338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757549.313341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757549.313346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757549.313350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757549.505301:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757549.505312:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757549.505322:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757549.505324:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757549.505421:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757549.505430:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757549.505438:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757550.337380:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757550.337393:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757550.337396:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757550.337403:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757550.337409:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757550.337415:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757550.337419:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757550.337423:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757550.337426:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757550.529421:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757550.529434:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757550.529445:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757550.529447:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757550.529582:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757550.529594:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757550.529602:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757551.361376:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757551.361389:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757551.361392:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757551.361399:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757551.361405:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757551.361412:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757551.361415:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757551.361420:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757551.361424:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757551.553356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757551.553366:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757551.553375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757551.553381:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757551.553387:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757551.553392:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757551.553399:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757551.553405:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757551.553408:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757551.553412:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757551.553489:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757551.553494:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757551.553498:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757552.385350:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757552.385359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757552.385361:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757552.385368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757552.385373:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757552.385377:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757552.385381:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757552.385384:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757552.385388:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757552.577301:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757552.577309:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757552.577317:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757552.577319:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757552.577437:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757552.577475:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757552.577480:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757552.833369:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757552.833382:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757552.833385:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757552.833390:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757552.833395:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757552.833405:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10607 00000100:00080000:2.0:1787757552.833419:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757552.833424:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757552.833426:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757552.833431:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757552.833460:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757552.833465:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757552.833470:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757552.833475:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757552.833478:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757552.833487:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757552.833493:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757552.833498:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757552.833505:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757552.833510:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833515:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757552.833520:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757552.833538:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757552.833542:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833547:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757552.833552:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757552.833559:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833577:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757552.833594:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659659392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757552.833612:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757552.833617:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833643:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757552.833647:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757552.833652:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757552.833654:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757552.833656:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757552.833671:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757552.833676:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10607 00000100:00080000:2.0:1787757552.833683:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757552.833686:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757552.833689:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757552.833692:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757552.833695:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757552.833698:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757552.833701:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757552.833707:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757552.833708:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757552.833710:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757552.833721:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757552.833722:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757552.833726:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.833729:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787757552.833733:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757552.833737:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787757552.833744:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757552.833745:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757552.833764:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.833768:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757552.833771:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757552.833776:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:2.0:1787757552.833779:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757552.833783:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757552.833788:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833794:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757552.833802:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659659392:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757552.833806:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757552.833814:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.833822:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757552.833823:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757552.833830:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757552.833834:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.833835:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757552.833845:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833850:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.833854:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:2.0:1787757552.833865:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659659520/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757552.833868:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833890:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757552.833894:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833901:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757552.833904:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757552.833909:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757552.833915:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757552.833916:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757552.833927:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757552.833929:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:2.0:1787757552.833933:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757552.833941:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10607 00000100:00080000:2.0:1787757552.833947:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757552.833949:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757552.833952:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757552.833955:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757552.833958:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757552.833960:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757552.833963:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757552.833966:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757552.833968:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757552.833970:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757552.833975:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757552.833978:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757552.833981:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757552.833985:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757552.833988:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.833992:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757552.833995:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757552.833999:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.834001:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757552.834007:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757552.834010:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.834014:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757552.834016:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757552.834019:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.834024:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757552.834030:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659659648/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757552.834045:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757552.834063:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757552.834065:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.834068:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757552.834072:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757552.834076:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757552.834080:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757552.834084:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757552.834092:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757552.834094:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:2.0:1787757552.834104:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10607 00000100:00000001:1.0:1787757552.834108:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:2.0:1787757552.834112:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757552.834114:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834120:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757552.834128:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757552.834132:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.834134:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757552.834155:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757552.834160:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305280705 00000020:00000001:2.0:1787757552.834160:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757552.834171:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757552.834177:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757552.834181:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757552.834232:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757552.834238:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757552.834243:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834249:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757552.834251:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757552.834257:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757552.834260:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.834262:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757552.834267:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757552.834273:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834276:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305280706 00000100:00000001:2.0:1787757552.834277:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834298:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757552.834299:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757552.834310:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.834316:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757552.834318:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757552.834321:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834324:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757552.834329:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834341:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757552.834343:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659659776/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757552.834356:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787757552.834361:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757552.834366:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757552.834371:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757552.834382:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757552/real 1787757552] req@ffff8ffb47a38380 x1874588659659392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757552.834410:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757552.834415:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757552.834428:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757552.834457:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757552.834461:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834465:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757552.834468:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757552.834477:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659659392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757552.834489:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834494:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757552.834498:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834501:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757552.834502:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834506:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757552.834509:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757552.834513:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757552.834520:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757552.834523:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757552.834529:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659659392:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757552.834537:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834542:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757552.834546:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757552.834551:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757552.834561:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757552.834566:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757552.834570:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757552.834578:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757552.834582:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757552.834589:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757552.834592:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834598:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834613:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757552.834617:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834622:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757552.834625:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757552.834628:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757552.834632:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757552.834635:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.834640:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757552.834644:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757552.834647:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757552.834652:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659520:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757552.834658:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757552.834661:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757552.834663:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.834666:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757552.834671:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834681:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757552.834683:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:1.0:1787757552.834690:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757552.834711:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757552.834719:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757552.834722:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757552.834724:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834726:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757552.834730:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834736:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305280014 00000100:00000001:1.0:1787757552.834740:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305280705 00000100:00000001:1.0:1787757552.834744:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305280014 00000100:00000001:1.0:1787757552.834746:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305280706 00000100:00000001:1.0:1787757552.834749:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757552.834752:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834755:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757552.834761:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757552/real 1787757552] req@ffff8ffb47a38000 x1874588659659520/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757552.834773:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757552.834775:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757552.834778:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757552.834780:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757552.834782:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834785:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757552.834787:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757552.834792:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659659520/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757552.834801:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834803:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757552.834806:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834808:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757552.834810:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.834812:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757552.834834:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757552.834838:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757552.834843:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757552.834846:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757552.834850:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659520:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757552.834855:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757552.834857:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757552.834859:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.834861:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757552.834863:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757552.834866:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757552.834870:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757552.834878:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757552.834880:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757552.834882:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.834886:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757552.834891:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834902:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757552.834905:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787757552.834913:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757552.834935:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757552.834944:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757552.834949:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757552.834952:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834955:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757552.834961:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834968:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305280011 00000100:00000001:1.0:1787757552.834973:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305280705 00000100:00000001:1.0:1787757552.834977:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305280011 00000100:00000001:1.0:1787757552.834982:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305280706 00000100:00000001:1.0:1787757552.834986:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757552.834990:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.834994:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757552.835000:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757552/real 1787757552] req@ffff8ffb47a3b480 x1874588659659648/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757552.835012:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757552.835014:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757552.835019:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757552.835021:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757552.835023:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835026:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757552.835028:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757552.835032:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659659648/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757552.835050:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835053:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757552.835056:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835058:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757552.835060:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835062:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757552.835082:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757552.835085:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757552.835089:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757552.835091:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757552.835095:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757552.835100:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757552.835102:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757552.835104:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.835108:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757552.835110:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757552.835113:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757552.835117:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757552.835121:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757552.835124:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757552.835126:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757552.835128:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757552.835132:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.835138:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757552.835140:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787757552.835146:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757552.835162:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757552.835168:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757552.835170:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757552.835172:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.835175:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757552.835179:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.835216:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305280011 00000100:00000001:1.0:1787757552.835221:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305280705 00000100:00000001:1.0:1787757552.835224:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305280012 00000100:00000001:1.0:1787757552.835227:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305280706 00000100:00000001:1.0:1787757552.835230:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757552.835232:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.835234:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757552.835239:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757552/real 1787757552] req@ffff8ffb47a39180 x1874588659659776/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757552.835249:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757552.835251:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757552.835255:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757552.835257:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757552.835259:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835261:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757552.835263:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757552.835268:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659659776/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757552.835276:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835279:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757552.835281:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835283:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757552.835285:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835287:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757552.835306:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757552.835309:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757552.835313:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757552.835316:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757552.835321:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659659776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757552.835327:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835329:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757552.835331:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757552.835334:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757552.835337:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757552.835338:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757552.835340:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757552.835345:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757552.835346:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757552.835349:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757552.835351:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835353:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757552.835355:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757552.835357:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757552.835359:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757552.835360:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757552.835362:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757552.835364:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757552.835365:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757552.835368:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757552.835370:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835372:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757552.835373:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757552.835374:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757552.835378:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757552.835379:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757552.835380:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757552.835382:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757552.835383:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757552.835387:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757552.835388:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835390:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757552.835395:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757552.835397:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757552.835399:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757552.835401:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757553.409399:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757553.409412:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757553.409415:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757553.409477:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757553.409484:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757553.409490:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757553.409495:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757553.409499:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757553.409504:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757553.601343:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757553.601354:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757553.601364:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757553.601366:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757553.601448:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757553.601454:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757553.601461:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757554.433347:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757554.433360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757554.433364:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757554.433372:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757554.433378:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757554.433384:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757554.433387:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757554.433391:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757554.433424:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757554.625369:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757554.625382:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757554.625387:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757554.625392:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757554.625401:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757554.625430:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757554.625433:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757554.625437:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757554.625443:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757554.625445:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757554.625567:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757554.625576:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757554.625583:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757555.457310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757555.457319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757555.457322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757555.457328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757555.457333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757555.457337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757555.457340:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757555.457344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757555.457348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757555.649297:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757555.649306:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757555.649314:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757555.649316:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757555.649344:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757555.649348:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757555.649353:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757556.481296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757556.481305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757556.481308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757556.481315:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757556.481319:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757556.481325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757556.481328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757556.481331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757556.481334:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757556.673296:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757556.673305:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757556.673313:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757556.673315:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757556.673469:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757556.673477:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757556.673484:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757557.505317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757557.505326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757557.505329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757557.505336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757557.505341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757557.505345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757557.505348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757557.505398:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757557.505403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757557.697325:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757557.697337:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757557.697348:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757557.697372:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757557.697406:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757557.697410:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757557.697413:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757557.953411:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757557.953429:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757557.953433:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757557.953438:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757557.953447:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757557.953458:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10612 00000100:00080000:3.0:1787757557.953477:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757557.953484:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757557.953488:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757557.953496:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757557.953502:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757557.953507:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757557.953513:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757557.953518:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757557.953523:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757557.953530:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757557.953536:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757557.953543:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757557.953550:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757557.953555:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.953558:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757557.953562:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757557.953578:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757557.953582:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.953588:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757557.953593:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757557.953600:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.953619:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757557.953636:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659659904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757557.953653:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757557.953660:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.953683:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757557.953688:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757557.953695:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757557.953705:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787757557.953710:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757557.953713:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757557.953717:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757557.953723:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787757557.953726:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757557.953736:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.953740:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757557.953752:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757557.953760:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10612 02000000:00000001:1.0:1787757557.953764:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757557.953771:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757557.953772:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.953788:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757557.953795:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757557.953797:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659659904:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757557.953811:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757557.953825:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757557.953832:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757557.953836:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:3.0:1787757557.953839:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757557.953845:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787757557.953848:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757557.953850:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757557.953859:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757557.953870:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757557.953871:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757557.953885:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.953887:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757557.953906:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757557.953923:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757557.953924:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757557.953931:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 02000000:00000001:3.0:1787757557.953934:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.953940:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757557.953944:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757557.953960:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757557.954003:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757557.954017:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757557.954023:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757557.954026:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954029:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757557.954036:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954041:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305285825 00000100:00000001:1.0:1787757557.954055:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757557.954062:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305285825 00000100:00000001:1.0:1787757557.954069:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757557.954075:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757557.954079:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954095:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00000001:3.0:1787757557.954095:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.954109:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757557.954118:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757557.954120:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.954123:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757557.954128:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757557.954130:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757557/real 1787757557] req@ffff8ffb45c31c00 x1874588659659904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757557.954133:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.954151:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757557.954160:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757557.954165:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:3.0:1787757557.954173:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659660032/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757557.954179:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757557.954267:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757557.954274:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.954282:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757557.954287:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757557.954300:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659659904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757557.954313:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.954319:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757557.954325:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.954328:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757557.954331:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.954335:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757557.954338:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757557.954368:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757557.954375:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757557.954379:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757557.954392:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659659904:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757557.954401:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.954404:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757557.954408:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757557.954411:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757557.954419:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757557.954423:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757557.954427:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757557.954432:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757557.954435:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757557.954438:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757557.954441:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.954451:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757557.954473:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757557.954475:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757557.954483:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954485:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954507:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757557.954512:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954514:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757557.954529:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757557.954539:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757557.954544:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.954551:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757557.954556:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757557.954560:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757557.954568:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660032:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757557.954575:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757557.954580:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757557.954584:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.954588:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757557.954594:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954606:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757557.954611:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757557.954618:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757557.954635:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757557.954644:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757557.954648:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757557.954651:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757557.954654:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757557.954664:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757557.954673:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757557.954674:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10612 00000100:00000001:1.0:1787757557.954680:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757557.954685:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757557.954687:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757557.954690:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757557.954695:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757557.954699:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.954699:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757557.954712:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757557.954714:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305285134 00000020:00000001:3.0:1787757557.954717:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757557.954731:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757557.954732:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305285825 00000020:00000001:3.0:1787757557.954741:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757557.954743:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305285134 00000100:00000001:3.0:1787757557.954763:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757557.954764:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305285825 00000100:00000001:3.0:1787757557.954772:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757557.954775:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757557.954787:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757557.954789:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757557.954804:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757557.954806:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757557.954812:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:1.0:1787757557.954820:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757557/real 1787757557] req@ffff8ffb45c33b80 x1874588659660032/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757557.954823:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757557.954843:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.954856:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000400:00000001:1.0:1787757557.954868:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000008:00100000:3.0:1787757557.954869:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:1.0:1787757557.954874:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000008:00000001:3.0:1787757557.954883:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954886:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757557.954888:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954892:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757557.954897:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757557.954898:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757557.954901:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757557.954905:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954907:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757557.954921:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:1.0:1787757557.954923:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659660032/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757557.954932:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757557.954948:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954949:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.954970:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757557.954974:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.954977:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757557.954985:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757557.954985:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.954998:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00100000:3.0:1787757557.955023:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659660160/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757557.955028:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757557.955048:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757557.955069:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00080000:1.0:1787757557.955074:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757557.955082:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955092:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757557.955096:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757557.955104:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660032:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1787757557.955105:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757557.955127:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757557.955130:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757557.955134:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757557.955141:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757557.955144:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757557.955146:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757557.955156:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787757557.955157:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757557.955170:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00080000:3.0:1787757557.955171:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10612 00000020:00000001:1.0:1787757557.955243:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00080000:3.0:1787757557.955245:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000020:00000001:1.0:1787757557.955255:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757557.955256:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757557.955265:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757557.955270:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000020:00000001:3.0:1787757557.955274:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757557.955278:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757557.955279:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000020:00000001:3.0:1787757557.955282:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757557.955287:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000020:00000001:3.0:1787757557.955288:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757557.955295:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757557.955299:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757557.955302:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757557.955304:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757557.955321:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:3.0:1787757557.955323:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757557.955329:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.955332:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955336:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757557.955338:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757557.955342:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757557.955369:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757557.955388:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757557.955388:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955397:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757557.955399:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757557.955404:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757557.955406:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757557.955412:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:3.0:1787757557.955415:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.955419:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757557.955421:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.955442:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757557.955448:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757557.955453:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757557.955455:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660160:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757557.955459:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757557.955468:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757557.955479:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757557.955481:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.955487:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757557.955493:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757557.955495:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.955497:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757557.955507:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757557.955512:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659660288/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757557.955522:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757557.955529:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757557.955531:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:3.0:1787757557.955541:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955549:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757557.955554:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757557.955582:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757557.955593:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757557.955598:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757557.955600:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955602:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757557.955606:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955611:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305285132 00000100:00000001:1.0:1787757557.955615:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305285825 00000100:00000001:1.0:1787757557.955618:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305285132 00000100:00000001:1.0:1787757557.955621:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305285825 00000100:00000001:1.0:1787757557.955624:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757557.955627:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955629:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757557.955636:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757557/real 1787757557] req@ffff8ffb45c30a80 x1874588659660160/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757557.955646:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757557.955648:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757557.955651:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757557.955653:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757557.955655:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955657:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757557.955659:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757557.955664:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659660160/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757557.955674:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955677:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757557.955680:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955682:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757557.955684:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955686:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757557.955701:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757557.955705:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757557.955709:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757557.955712:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757557.955716:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660160:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757557.955721:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955723:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757557.955725:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757557.955727:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757557.955731:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757557.955732:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757557.955734:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757557.955737:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757557.955740:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757557.955742:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757557.955744:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955746:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955751:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757557.955753:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955757:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757557.955759:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757557.955761:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757557.955763:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757557.955765:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.955769:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757557.955771:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757557.955774:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757557.955778:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757557.955782:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757557.955785:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757557.955787:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757557.955790:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757557.955793:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955798:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757557.955800:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757557.955804:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757557.955818:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757557.955823:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757557.955826:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757557.955828:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955830:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757557.955833:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955837:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305285132 00000100:00000001:1.0:1787757557.955840:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305285825 00000100:00000001:1.0:1787757557.955843:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305285132 00000100:00000001:1.0:1787757557.955846:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305285825 00000100:00000001:1.0:1787757557.955848:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757557.955851:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955853:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757557.955858:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757557/real 1787757557] req@ffff8ffb45c33b80 x1874588659660288/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757557.955867:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757557.955868:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757557.955871:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757557.955873:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757557.955875:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955877:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757557.955879:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757557.955883:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659660288/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757557.955900:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955903:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757557.955906:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955908:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757557.955910:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955912:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757557.955927:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757557.955930:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757557.955934:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757557.955936:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757557.955940:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757557.955945:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955947:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757557.955949:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757557.955950:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757557.955953:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757557.955954:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757557.955956:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757557.955958:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757557.955959:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757557.955961:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757557.955963:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955965:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757557.955968:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757557.955970:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757557.955972:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757557.955975:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757558.529390:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757558.529404:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757558.529407:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757558.529416:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757558.529421:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757558.529428:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757558.529432:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757558.529436:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757558.529440:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757558.721370:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757558.721383:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757558.721395:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757558.721398:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757558.721460:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757558.721467:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757558.721473:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757559.553473:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757559.553493:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757559.553498:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757559.553513:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757559.553526:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757559.553535:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757559.553540:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757559.553546:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757559.553552:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757559.745396:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757559.745409:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757559.745422:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757559.745426:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757559.745468:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757559.745472:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757559.745476:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757560.577285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757560.577329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757560.577332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757560.577339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757560.577343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757560.577348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757560.577352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757560.577355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757560.577358:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757560.769346:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757560.769356:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757560.769366:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757560.769368:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757560.769472:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757560.769478:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757560.769483:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757561.601279:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757561.601337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757561.601340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757561.601348:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757561.601352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757561.601357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757561.601361:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757561.601364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757561.601368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757561.793402:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757561.793410:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757561.793417:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757561.793422:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757561.793425:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757561.793430:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757561.793434:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757561.793437:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757561.793440:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757561.793443:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757561.793466:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757561.793469:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757561.793473:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757562.625340:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757562.625349:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757562.625351:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757562.625358:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757562.625364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757562.625371:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757562.625376:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757562.625380:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757562.625385:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757562.817376:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757562.817384:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757562.817393:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757562.817396:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757562.817475:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757562.817481:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757562.817486:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757563.073330:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757563.073342:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757563.073345:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757563.073350:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757563.073356:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757563.073364:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10617 00000100:00080000:3.0:1787757563.073378:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757563.073382:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757563.073385:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757563.073390:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757563.073393:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757563.073396:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757563.073399:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757563.073402:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757563.073404:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757563.073410:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757563.073413:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.073416:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757563.073421:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757563.073425:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073427:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757563.073430:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.073444:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757563.073446:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073450:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757563.073453:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757563.073458:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073475:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757563.073487:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659660416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757563.073500:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757563.073505:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073528:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757563.073532:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757563.073537:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757563.073539:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757563.073541:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757563.073545:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757563.073549:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10617 00000100:00080000:3.0:1787757563.073554:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757563.073556:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757563.073558:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757563.073561:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757563.073563:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757563.073565:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757563.073568:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757563.073571:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757563.073572:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757563.073576:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757563.073579:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.073581:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757563.073587:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757563.073590:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073592:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757563.073599:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757563.073603:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073606:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.073612:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757563.073614:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073616:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757563.073618:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757563.073621:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.073627:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757563.073628:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.073638:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:3.0:1787757563.073641:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659660544/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757563.073645:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757563.073652:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757563.073657:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073660:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757563.073666:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757563.073667:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757563.073674:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757563.073675:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757563.073685:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757563.073687:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757563.073699:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787757563.073708:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659660416:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757563.073717:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757563.073725:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757563.073727:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757563.073731:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757563.073738:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10617 00000100:00080000:3.0:1787757563.073743:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757563.073745:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757563.073747:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757563.073750:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757563.073752:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757563.073756:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787757563.073756:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757563.073765:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787757563.073766:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073769:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757563.073777:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757563.073779:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757563.073786:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757563.073791:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.073794:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757563.073795:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.073798:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757563.073805:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073808:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757563.073810:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757563.073814:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073817:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.073824:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757563.073826:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.073828:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757563.073829:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757563.073837:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 02000000:00000001:3.0:1787757563.073838:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757563.073844:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.073848:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757563.073853:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757563.073863:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659660672/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757563.073874:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757563.073877:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073881:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757563.073884:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757563.073890:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757563.073892:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757563.073894:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757563.073897:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757563.073901:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10617 00000800:00000100:1.0:1787757563.073910:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757563.073912:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757563.073918:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757563.073921:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757563.073925:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757563.073927:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757563.073929:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:3.0:1787757563.073931:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757563.073938:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757563.073940:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757563.073942:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757563.073947:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073948:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757563.073958:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757563.073964:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757563.073971:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.073972:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.073977:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757563.073982:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305290945 02000000:00000001:3.0:1787757563.073984:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757563.073991:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757563.073994:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757563.073996:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757563.073999:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074001:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757563.074002:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074016:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305290945 02000000:00000001:3.0:1787757563.074017:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757563.074028:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757563.074031:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074035:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757563.074037:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757563.074042:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757563.074045:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757563.074047:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074053:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757563.074060:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757563.074067:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757563/real 1787757563] req@ffff8ffb45c31c00 x1874588659660416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757563.074076:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659660800/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757563.074119:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:1.0:1787757563.074120:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757563.074125:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757563.074126:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757563.074135:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074137:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757563.074145:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757563.074149:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074153:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757563.074155:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757563.074164:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659660416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757563.074176:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074228:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757563.074234:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074236:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757563.074237:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074241:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757563.074243:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757563.074247:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757563.074252:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757563.074285:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757563.074291:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659660416:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757563.074298:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074301:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757563.074303:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757563.074306:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757563.074313:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757563.074316:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757563.074318:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757563.074323:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757563.074325:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757563.074328:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757563.074330:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074334:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074343:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757563.074345:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074348:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757563.074350:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757563.074352:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757563.074355:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757563.074357:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757563.074360:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757563.074363:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757563.074366:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757563.074370:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660544:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757563.074374:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757563.074377:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757563.074379:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757563.074381:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757563.074385:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074395:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757563.074396:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757563.074402:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757563.074420:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757563.074426:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757563.074430:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757563.074431:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074434:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757563.074437:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074443:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305290254 00000100:00000001:1.0:1787757563.074447:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305290945 00000100:00000001:1.0:1787757563.074450:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305290254 00000100:00000001:1.0:1787757563.074453:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305290945 00000100:00000001:1.0:1787757563.074456:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757563.074458:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074461:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757563.074466:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757563/real 1787757563] req@ffff8ffb45c30e00 x1874588659660544/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757563.074476:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757563.074478:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757563.074481:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757563.074483:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757563.074485:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074488:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757563.074489:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757563.074494:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659660544/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757563.074503:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074505:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757563.074508:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074510:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757563.074512:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074514:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757563.074529:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757563.074532:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757563.074536:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757563.074538:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757563.074542:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660544:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757563.074546:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757563.074548:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757563.074550:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757563.074553:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757563.074555:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757563.074557:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757563.074561:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660672:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757563.074565:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757563.074566:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757563.074568:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757563.074570:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757563.074574:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074580:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757563.074581:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787757563.074586:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757563.074599:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757563.074603:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757563.074606:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757563.074608:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074610:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757563.074613:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074617:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305290251 00000100:00000001:1.0:1787757563.074620:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305290945 00000100:00000001:1.0:1787757563.074623:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305290251 00000100:00000001:1.0:1787757563.074625:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305290945 00000100:00000001:1.0:1787757563.074628:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757563.074630:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074632:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757563.074637:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757563/real 1787757563] req@ffff8ffb45c30000 x1874588659660672/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757563.074646:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757563.074648:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757563.074650:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757563.074652:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757563.074654:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074656:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757563.074658:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757563.074662:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659660672/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757563.074671:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074673:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757563.074676:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074678:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757563.074680:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074681:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757563.074700:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757563.074705:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757563.074713:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757563.074717:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757563.074723:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660672:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757563.074730:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757563.074732:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757563.074735:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757563.074739:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757563.074742:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757563.074747:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757563.074753:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757563.074760:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757563.074763:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757563.074766:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757563.074770:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757563.074776:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074786:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757563.074789:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757563.074796:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757563.074817:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757563.074823:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757563.074827:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757563.074829:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074831:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757563.074834:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074839:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305290251 00000100:00000001:1.0:1787757563.074842:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305290945 00000100:00000001:1.0:1787757563.074845:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305290251 00000100:00000001:1.0:1787757563.074848:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305290945 00000100:00000001:1.0:1787757563.074851:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757563.074853:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.074855:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757563.074861:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757563/real 1787757563] req@ffff8ffb45c30700 x1874588659660800/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757563.074871:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757563.074873:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757563.074876:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757563.074878:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757563.074880:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074882:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757563.074884:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757563.074889:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659660800/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757563.074897:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074899:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757563.074902:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074904:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757563.074906:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074908:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757563.074920:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757563.074923:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757563.074927:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757563.074929:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757563.074933:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659660800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757563.074937:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074939:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757563.074941:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757563.074943:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757563.074946:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757563.074948:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757563.074950:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757563.074953:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757563.074955:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757563.074958:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757563.074960:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074962:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757563.074964:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757563.074966:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757563.074969:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757563.074971:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757563.074972:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757563.074974:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757563.074975:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757563.074977:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757563.074978:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074980:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757563.074981:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757563.074983:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757563.074984:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757563.074986:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757563.074987:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757563.074989:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757563.074990:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757563.074992:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757563.074993:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.074995:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757563.075000:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757563.075001:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757563.075004:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757563.075006:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757563.649423:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757563.649437:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757563.649440:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757563.649448:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757563.649453:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757563.649459:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757563.649463:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757563.649468:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757563.649471:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757563.841395:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757563.841406:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757563.841416:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757563.841418:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757563.841455:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757563.841459:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757563.841463:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757564.673371:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757564.673387:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757564.673392:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757564.673404:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757564.673411:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757564.673419:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757564.673424:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757564.673431:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757564.673437:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757564.865402:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757564.865417:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757564.865422:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757564.865427:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757564.865436:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757564.865440:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757564.865444:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757564.865447:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757564.865455:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757564.865458:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757564.865594:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757564.865605:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757564.865613:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757565.697347:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757565.697360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757565.697363:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757565.697372:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757565.697378:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757565.697384:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757565.697388:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757565.697392:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757565.697396:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757565.889380:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757565.889390:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757565.889399:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757565.889403:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757565.889496:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757565.889501:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757565.889507:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757566.721327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757566.721343:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757566.721348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757566.721359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757566.721366:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757566.721374:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757566.721380:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757566.721386:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757566.721393:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757566.913319:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757566.913329:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757566.913340:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757566.913343:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757566.913453:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757566.913463:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757566.913470:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757567.745355:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757567.745366:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757567.745369:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757567.745377:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757567.745383:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757567.745389:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757567.745392:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757567.745397:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757567.745401:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757567.937342:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757567.937351:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757567.937359:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757567.937361:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757567.937391:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757567.937394:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757567.937398:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757568.193282:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757568.193296:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757568.193300:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757568.193305:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757568.193312:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757568.193324:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10622 00000100:00080000:3.0:1787757568.193341:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757568.193346:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757568.193349:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757568.193356:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757568.193360:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757568.193364:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757568.193368:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757568.193371:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757568.193374:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757568.193380:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757568.193384:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757568.193387:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757568.193392:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757568.193397:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193401:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757568.193404:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757568.193419:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757568.193422:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193427:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757568.193432:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757568.193439:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193457:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757568.193471:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659660928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757568.193487:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757568.193493:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193514:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757568.193519:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757568.193524:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757568.193526:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757568.193528:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757568.193533:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757568.193536:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10622 00000100:00080000:3.0:1787757568.193540:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757568.193544:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757568.193546:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757568.193550:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757568.193552:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757568.193555:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757568.193559:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757568.193561:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757568.193563:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757568.193568:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757568.193571:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757568.193573:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757568.193581:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757568.193582:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757568.193590:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193593:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757568.193594:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787757568.193600:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757568.193601:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757568.193607:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:3.0:1787757568.193611:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193613:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193625:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.193628:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757568.193636:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757568.193640:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757568.193641:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757568.193649:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757568.193656:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659660928:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757568.193658:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757568.193666:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757568.193668:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757568.193675:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.193678:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757568.193686:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.193691:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757568.193693:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.193700:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757568.193714:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659661056/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757568.193729:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757568.193733:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757568.193734:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193740:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757568.193745:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193749:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757568.193752:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757568.193759:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757568.193761:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757568.193763:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757568.193769:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757568.193774:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10622 00000100:00080000:3.0:1787757568.193781:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757568.193783:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757568.193785:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757568.193788:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757568.193791:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757568.193795:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757568.193798:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757568.193801:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757568.193803:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757568.193808:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757568.193811:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757568.193813:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757568.193817:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757568.193821:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193824:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757568.193829:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757568.193838:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:3.0:1787757568.193839:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193847:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757568.193855:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757568.193858:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193861:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757568.193863:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757568.193864:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757568.193872:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193875:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757568.193881:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193882:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193894:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:3.0:1787757568.193900:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659661184/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757568.193905:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193918:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305296065 00000100:00000001:3.0:1787757568.193920:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757568.193927:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193938:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757568.193950:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.193951:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305296065 00000100:00080000:3.0:1787757568.193956:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757568.193962:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787757568.193963:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757568.193971:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757568.193973:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757568.193977:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757568.193981:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10622 00000100:00000001:1.0:1787757568.193983:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757568.193987:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757568.193990:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.193994:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757568.193997:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757568.194000:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757568.194006:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757568.194010:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:1.0:1787757568.194013:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757568/real 1787757568] req@ffff8ffb45c32a00 x1874588659660928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757568.194014:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757568.194023:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757568.194032:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757568.194036:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757568.194049:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757568.194057:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757568.194058:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787757568.194062:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000400:00000001:1.0:1787757568.194064:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757568.194068:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757568.194073:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787757568.194074:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.194081:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757568.194082:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000008:00100000:3.0:1787757568.194085:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757568.194088:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787757568.194091:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.194097:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194098:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757568.194104:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757568.194107:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757568.194109:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757568.194118:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757568.194121:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757568.194124:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757568.194131:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659660928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757568.194133:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757568.194148:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659661312/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757568.194245:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757568.194252:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194253:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757568.194259:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194263:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757568.194272:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194275:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757568.194278:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194282:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757568.194287:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757568.194293:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757568.194301:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757568.194306:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757568.194311:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659660928:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757568.194319:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194323:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757568.194325:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757568.194329:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757568.194336:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757568.194340:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757568.194343:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757568.194348:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757568.194351:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757568.194355:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757568.194357:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194360:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194371:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757568.194374:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194377:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757568.194379:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757568.194381:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757568.194383:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757568.194386:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.194389:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757568.194392:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757568.194394:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757568.194398:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661056:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757568.194403:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757568.194419:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757568.194421:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.194424:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757568.194428:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194438:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757568.194440:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757568.194447:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757568.194467:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757568.194474:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757568.194477:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757568.194479:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194482:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757568.194485:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194492:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305295374 00000100:00000001:1.0:1787757568.194497:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305296065 00000100:00000001:1.0:1787757568.194500:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305295374 00000100:00000001:1.0:1787757568.194503:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305296065 00000100:00000001:1.0:1787757568.194506:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757568.194508:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194511:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757568.194517:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757568/real 1787757568] req@ffff8ffb45c31500 x1874588659661056/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757568.194528:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757568.194530:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757568.194533:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757568.194535:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757568.194537:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194540:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757568.194542:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757568.194547:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659661056/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757568.194556:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194558:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757568.194561:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194563:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757568.194564:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194566:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757568.194588:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757568.194593:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757568.194597:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757568.194600:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757568.194604:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661056:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757568.194609:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757568.194611:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757568.194613:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.194615:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757568.194617:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757568.194620:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757568.194624:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661184:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757568.194628:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757568.194630:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757568.194632:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.194634:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757568.194638:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194644:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757568.194646:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757568.194651:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757568.194663:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757568.194667:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757568.194670:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757568.194672:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194675:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757568.194678:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194682:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305295371 00000100:00000001:1.0:1787757568.194685:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305296065 00000100:00000001:1.0:1787757568.194688:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305295371 00000100:00000001:1.0:1787757568.194691:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305296065 00000100:00000001:1.0:1787757568.194694:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757568.194696:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194699:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757568.194704:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757568/real 1787757568] req@ffff8ffb45c33800 x1874588659661184/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757568.194714:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757568.194716:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757568.194718:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757568.194720:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757568.194722:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194725:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757568.194727:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757568.194731:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659661184/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757568.194739:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194742:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757568.194745:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194746:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757568.194748:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194750:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757568.194768:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757568.194771:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757568.194774:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757568.194777:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757568.194780:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661184:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757568.194785:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757568.194786:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757568.194788:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.194791:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757568.194793:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757568.194795:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757568.194799:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757568.194807:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757568.194809:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757568.194811:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757568.194813:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757568.194816:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194821:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757568.194823:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757568.194828:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757568.194839:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757568.194844:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757568.194847:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757568.194849:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194851:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757568.194854:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194858:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305295371 00000100:00000001:1.0:1787757568.194861:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305296065 00000100:00000001:1.0:1787757568.194864:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305295371 00000100:00000001:1.0:1787757568.194867:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305296065 00000100:00000001:1.0:1787757568.194870:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757568.194872:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.194874:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757568.194879:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757568/real 1787757568] req@ffff8ffb45c31180 x1874588659661312/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757568.194888:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757568.194889:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757568.194891:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757568.194893:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757568.194895:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194897:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757568.194898:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757568.194902:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659661312/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757568.194919:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194922:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757568.194925:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194927:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757568.194929:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194931:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757568.194949:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757568.194952:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757568.194956:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757568.194958:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757568.194961:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757568.194966:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194969:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757568.194971:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757568.194973:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757568.194976:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757568.194977:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757568.194979:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757568.194981:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757568.194983:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757568.194986:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757568.194987:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.194990:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757568.194991:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757568.194993:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757568.194995:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757568.194996:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757568.194997:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757568.194999:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757568.195001:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757568.195003:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757568.195004:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.195006:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757568.195007:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757568.195010:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757568.195014:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757568.195015:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757568.195017:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757568.195019:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757568.195020:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757568.195022:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757568.195024:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.195026:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757568.195030:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757568.195032:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757568.195034:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757568.195037:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757568.769295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757568.769307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757568.769310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757568.769317:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757568.769323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757568.769329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757568.769333:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757568.769337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757568.769341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757568.961328:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757568.961338:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757568.961350:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757568.961353:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757568.961436:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757568.961443:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757568.961449:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757569.793315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757569.793329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757569.793332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757569.793340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757569.793346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757569.793352:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757569.793355:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757569.793359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757569.793363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757569.985323:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757569.985346:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757569.985356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757569.985357:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757569.985391:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757569.985395:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757569.985400:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757570.817316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757570.817325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757570.817328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757570.817334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757570.817339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757570.817344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757570.817349:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757570.817354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757570.817358:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757571.009329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757571.009337:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757571.009357:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757571.009359:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757571.009467:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757571.009481:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757571.009492:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757571.841307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757571.841316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757571.841319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757571.841325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757571.841330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757571.841336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757571.841340:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757571.841343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757571.841347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757572.033368:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757572.033376:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757572.033388:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757572.033395:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757572.033399:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757572.033403:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757572.033406:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757572.033409:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757572.033411:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757572.033414:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757572.033443:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757572.033446:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757572.033450:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757572.865299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757572.865308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757572.865311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757572.865318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757572.865323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757572.865327:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757572.865331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757572.865334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757572.865338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757573.057298:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757573.057306:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757573.057315:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757573.057317:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757573.057433:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757573.057440:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757573.057445:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757573.313335:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757573.313347:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757573.313350:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757573.313355:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757573.313360:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757573.313370:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10628 00000100:00080000:2.0:1787757573.313384:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757573.313388:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757573.313391:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757573.313396:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757573.313399:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757573.313403:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757573.313406:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757573.313409:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757573.313411:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757573.313417:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757573.313420:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757573.313424:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757573.313428:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757573.313432:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313434:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757573.313437:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757573.313449:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757573.313451:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313456:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757573.313459:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757573.313465:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313478:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757573.313490:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659661440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757573.313504:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757573.313509:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313532:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757573.313537:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757573.313541:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757573.313543:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757573.313545:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757573.313548:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757573.313551:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10628 00000100:00080000:2.0:1787757573.313556:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757573.313559:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757573.313562:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757573.313565:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757573.313567:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757573.313570:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757573.313575:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757573.313578:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757573.313579:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757573.313584:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757573.313587:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757573.313590:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757573.313593:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757573.313597:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757573.313602:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757573.313603:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.313609:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757573.313610:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757573.313618:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:2.0:1787757573.313621:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757573.313623:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757573.313633:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.313637:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757573.313639:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.313644:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757573.313647:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757573.313651:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757573.313652:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313660:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757573.313664:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757573.313667:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659661440:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757573.313669:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.313682:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757573.313683:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.313689:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757573.313694:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757573.313697:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659661568/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757573.313699:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757573.313710:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313714:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757573.313719:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313724:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757573.313729:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757573.313733:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757573.313735:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757573.313737:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757573.313743:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00080000:2.0:1787757573.313745:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757573.313752:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757573.313756:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757573.313766:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10628 00000100:00080000:2.0:1787757573.313775:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757573.313779:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757573.313781:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757573.313785:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757573.313788:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757573.313791:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757573.313796:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757573.313801:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757573.313805:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757573.313813:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1787757573.313823:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757573.313823:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757573.313833:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757573.313844:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757573.313845:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757573.313853:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787757573.313855:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.313859:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313864:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757573.313865:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:2.0:1787757573.313874:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757573.313877:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757573.313886:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.313888:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305301185 00000100:00000001:1.0:1787757573.313901:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757573.313909:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305301185 00000100:00000001:1.0:1787757573.313919:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757573.313921:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757573.313935:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757573.313937:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757573.313940:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.313946:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.313948:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757573.313954:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757573.313957:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757573.313960:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757573.313971:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757573/real 1787757573] req@ffff8ffb47a38700 x1874588659661440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757573.313974:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757573.313994:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659661696/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757573.313997:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757573.314005:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757573.314014:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757573.314018:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757573.314019:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.314029:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314030:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:2.0:1787757573.314033:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757573.314038:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757573.314040:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757573.314046:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757573.314051:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:2.0:1787757573.314052:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757573.314061:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757573.314061:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757573.314066:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10628 00000100:00080000:2.0:1787757573.314072:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757573.314101:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757573.314113:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757573.314119:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757573.314122:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757573.314128:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757573.314136:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:1.0:1787757573.314143:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659661440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757573.314145:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757573.314153:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757573.314171:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757573.314172:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757573.314181:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757573.314228:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314230:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757573.314233:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314237:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757573.314239:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757573.314244:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757573.314245:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:2.0:1787757573.314253:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00080000:1.0:1787757573.314260:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:2.0:1787757573.314263:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757573.314267:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:2.0:1787757573.314270:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757573.314276:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:1.0:1787757573.314279:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659661440:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1787757573.314282:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757573.314293:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314297:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757573.314299:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314309:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787757573.314313:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757573.314315:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757573.314320:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.314323:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:2.0:1787757573.314327:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757573.314331:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757573.314332:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:2.0:1787757573.314335:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757573.314339:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757573.314344:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:2.0:1787757573.314347:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.314350:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757573.314356:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757573.314360:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00100000:2.0:1787757573.314362:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659661824/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757573.314365:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757573.314380:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757573.314389:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314390:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757573.314402:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314408:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757573.314414:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314419:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757573.314421:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757573.314424:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757573.314426:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757573.314429:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.314433:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757573.314437:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757573.314440:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757573.314446:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661568:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757573.314451:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757573.314454:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757573.314456:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.314459:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757573.314463:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314473:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757573.314475:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:1.0:1787757573.314481:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757573.314501:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757573.314508:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757573.314511:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757573.314513:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314515:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757573.314519:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314525:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305300494 00000100:00000001:1.0:1787757573.314529:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305301185 00000100:00000001:1.0:1787757573.314532:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305300494 00000100:00000001:1.0:1787757573.314535:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305301185 00000100:00000001:1.0:1787757573.314538:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757573.314541:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314543:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757573.314549:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757573/real 1787757573] req@ffff8ffb47a3a300 x1874588659661568/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757573.314562:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757573.314564:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757573.314567:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757573.314569:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757573.314571:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314573:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757573.314575:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757573.314580:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659661568/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757573.314589:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314592:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757573.314596:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314598:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757573.314599:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314601:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757573.314617:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757573.314620:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757573.314624:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757573.314627:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757573.314631:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661568:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757573.314636:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757573.314638:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757573.314639:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.314642:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757573.314644:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757573.314647:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757573.314651:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757573.314655:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757573.314657:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757573.314659:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.314662:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757573.314665:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314671:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757573.314672:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787757573.314677:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757573.314689:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757573.314694:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757573.314696:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757573.314698:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314701:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757573.314704:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314707:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305300491 00000100:00000001:1.0:1787757573.314710:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305301185 00000100:00000001:1.0:1787757573.314713:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305300491 00000100:00000001:1.0:1787757573.314716:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305301185 00000100:00000001:1.0:1787757573.314719:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757573.314721:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314723:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757573.314728:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757573/real 1787757573] req@ffff8ffb47a39180 x1874588659661696/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757573.314738:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757573.314739:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757573.314742:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757573.314744:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757573.314746:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314748:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757573.314749:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757573.314754:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659661696/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757573.314771:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314774:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757573.314777:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314779:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757573.314781:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314783:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757573.314797:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757573.314800:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757573.314803:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757573.314805:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757573.314810:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757573.314817:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757573.314819:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757573.314822:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.314826:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757573.314829:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757573.314833:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757573.314839:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661824:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757573.314845:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757573.314849:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757573.314851:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757573.314855:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757573.314860:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314869:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757573.314872:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787757573.314878:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757573.314901:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757573.314910:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757573.314914:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757573.314918:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314921:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757573.314926:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314932:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305300491 00000100:00000001:1.0:1787757573.314937:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305301185 00000100:00000001:1.0:1787757573.314942:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305300491 00000100:00000001:1.0:1787757573.314946:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305301185 00000100:00000001:1.0:1787757573.314950:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757573.314954:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.314958:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757573.314966:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757573/real 1787757573] req@ffff8ffb47a3b480 x1874588659661824/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757573.314981:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757573.314983:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757573.314988:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757573.314990:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757573.314993:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.314996:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757573.314999:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757573.315007:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659661824/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757573.315018:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.315022:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757573.315026:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.315029:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757573.315031:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.315034:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757573.315055:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757573.315061:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757573.315068:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757573.315096:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757573.315105:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659661824:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757573.315113:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.315117:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757573.315119:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757573.315122:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757573.315127:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757573.315130:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757573.315133:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757573.315137:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757573.315139:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757573.315145:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757573.315147:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.315151:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757573.315154:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757573.315156:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757573.315161:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757573.315163:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757573.315166:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757573.315170:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757573.315173:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757573.315177:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757573.315179:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.315183:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757573.315219:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757573.315223:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757573.315227:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757573.315230:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757573.315232:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757573.315237:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757573.315239:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757573.315243:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757573.315246:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.315250:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757573.315256:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757573.315258:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757573.315261:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757573.315263:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757573.889295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757573.889306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757573.889311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757573.889320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757573.889325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757573.889331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757573.889336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757573.889341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757573.889347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757574.081311:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757574.081320:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757574.081329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757574.081332:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757574.081378:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757574.081385:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757574.081391:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757574.913300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757574.913309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757574.913313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757574.913319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757574.913323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757574.913328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757574.913331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757574.913335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757574.913338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757575.105296:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757575.105306:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757575.105310:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757575.105314:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757575.105322:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757575.105325:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757575.105328:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757575.105331:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757575.105337:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757575.105340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757575.105423:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757575.105428:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757575.105432:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757575.937319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757575.937328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757575.937331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757575.937338:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757575.937344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757575.937349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757575.937353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757575.937356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757575.937360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757576.129309:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757576.129318:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757576.129327:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757576.129330:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757576.129356:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757576.129361:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757576.129367:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757576.961301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757576.961313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757576.961317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757576.961324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757576.961330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757576.961335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757576.961339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757576.961343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757576.961347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757577.153332:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757577.153342:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757577.153350:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757577.153352:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757577.153464:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757577.153490:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757577.153507:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757577.985321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757577.985332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757577.985336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757577.985345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757577.985350:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757577.985356:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757577.985359:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757577.985363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757577.985367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757578.177329:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757578.177340:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757578.177349:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757578.177351:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757578.177478:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757578.177485:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757578.177492:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757578.433391:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757578.433407:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757578.433411:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757578.433417:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757578.433424:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757578.433436:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10633 00000100:00080000:3.0:1787757578.433454:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757578.433458:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757578.433461:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757578.433468:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757578.433472:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757578.433476:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757578.433479:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757578.433482:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757578.433484:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757578.433491:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757578.433494:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757578.433498:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757578.433503:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757578.433508:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433511:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757578.433514:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757578.433532:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757578.433534:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433539:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757578.433543:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757578.433551:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433570:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757578.433585:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659661952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757578.433601:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757578.433607:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433634:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757578.433640:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757578.433645:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757578.433647:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757578.433649:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757578.433653:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757578.433656:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10633 00000100:00080000:3.0:1787757578.433660:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757578.433663:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757578.433665:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757578.433669:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757578.433671:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757578.433674:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757578.433677:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757578.433680:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757578.433682:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757578.433687:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757578.433690:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757578.433692:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757578.433696:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757578.433700:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757578.433706:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757578.433707:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.433714:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757578.433714:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757578.433722:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:3.0:1787757578.433725:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757578.433727:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757578.433732:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433738:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757578.433739:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757578.433745:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757578.433746:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757578.433754:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757578.433755:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433766:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757578.433770:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757578.433774:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659661952:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757578.433775:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433787:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.433788:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757578.433798:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:3.0:1787757578.433802:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659662080/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757578.433804:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757578.433814:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757578.433825:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433840:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757578.433844:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433848:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757578.433852:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757578.433856:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757578.433857:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757578.433860:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757578.433861:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757578.433864:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757578.433866:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00080000:3.0:1787757578.433873:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10633 00000100:00000001:1.0:1787757578.433878:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757578.433881:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757578.433886:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757578.433888:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757578.433890:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757578.433892:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757578.433895:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757578.433898:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757578.433900:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757578.433903:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757578.433907:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757578.433910:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757578.433912:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757578.433916:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757578.433918:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433921:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757578.433925:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757578.433929:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433931:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757578.433937:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757578.433939:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433946:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000800:00000100:1.0:1787757578.433947:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757578.433950:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757578.433956:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.433962:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757578.433969:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659662208/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757578.433976:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757578.434007:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757578.434012:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757578.434014:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434023:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.434024:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434035:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757578.434038:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757578.434048:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757578.434053:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757578.434054:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757578.434058:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757578.434068:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305306305 00000100:00000001:3.0:1787757578.434069:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757578.434080:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10633 00000100:00080000:3.0:1787757578.434089:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757578.434093:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757578.434097:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757578.434101:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757578.434105:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757578.434106:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787757578.434111:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757578.434119:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305306305 00000020:00000001:3.0:1787757578.434121:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757578.434132:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757578.434133:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757578.434140:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757578.434142:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757578.434153:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757578.434158:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757578.434160:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757578.434168:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757578.434173:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:1.0:1787757578.434176:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757578/real 1787757578] req@ffff8ffb45c33480 x1874588659661952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757578.434180:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757578.434232:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.434236:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757578.434239:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757578.434243:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.434246:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757578.434253:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757578.434256:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.434264:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757578.434269:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757578.434274:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757578.434287:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757578.434291:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00100000:3.0:1787757578.434299:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659662336/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757578.434301:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757578.434315:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757578.434317:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757578.434321:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434325:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757578.434328:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434333:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434340:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757578.434344:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757578.434355:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659661952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757578.434367:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434373:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757578.434377:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434379:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757578.434381:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434384:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757578.434387:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757578.434393:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757578.434401:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757578.434406:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757578.434414:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659661952:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757578.434424:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434429:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757578.434433:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757578.434438:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757578.434447:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757578.434451:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757578.434455:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757578.434462:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757578.434466:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757578.434472:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757578.434475:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434480:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434493:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757578.434495:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434499:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757578.434501:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757578.434504:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757578.434506:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757578.434508:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757578.434512:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757578.434515:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757578.434519:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757578.434524:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662080:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757578.434529:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757578.434532:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757578.434534:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757578.434537:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757578.434542:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434551:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757578.434553:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757578.434560:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757578.434580:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757578.434588:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757578.434591:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757578.434593:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434595:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757578.434600:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434605:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305305614 00000100:00000001:1.0:1787757578.434610:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305306305 00000100:00000001:1.0:1787757578.434613:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305305614 00000100:00000001:1.0:1787757578.434616:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305306305 00000100:00000001:1.0:1787757578.434619:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757578.434622:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434624:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757578.434629:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757578/real 1787757578] req@ffff8ffb45c31180 x1874588659662080/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757578.434640:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757578.434642:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757578.434645:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757578.434647:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757578.434649:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434651:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757578.434653:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757578.434658:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659662080/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757578.434668:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434672:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757578.434675:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434677:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757578.434678:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434680:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757578.434702:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757578.434705:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757578.434708:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757578.434711:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757578.434715:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662080:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757578.434720:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757578.434722:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757578.434724:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757578.434727:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757578.434729:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757578.434732:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757578.434735:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757578.434739:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757578.434741:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757578.434743:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757578.434745:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757578.434749:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434755:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757578.434757:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757578.434763:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757578.434775:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757578.434781:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757578.434783:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757578.434786:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434788:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757578.434791:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434795:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305305611 00000100:00000001:1.0:1787757578.434798:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305306305 00000100:00000001:1.0:1787757578.434801:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305305611 00000100:00000001:1.0:1787757578.434804:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305306305 00000100:00000001:1.0:1787757578.434807:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757578.434809:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434811:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757578.434816:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757578/real 1787757578] req@ffff8ffb45c33800 x1874588659662208/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757578.434825:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757578.434827:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757578.434830:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757578.434832:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757578.434834:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434836:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757578.434838:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757578.434842:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659662208/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757578.434851:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434853:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757578.434856:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434858:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757578.434859:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.434861:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757578.434880:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757578.434883:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757578.434887:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757578.434890:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757578.434895:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757578.434900:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757578.434901:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757578.434903:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757578.434906:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757578.434908:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757578.434910:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757578.434913:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757578.434917:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757578.434919:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757578.434920:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757578.434922:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757578.434926:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434930:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757578.434932:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757578.434937:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757578.434948:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757578.434953:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757578.434956:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757578.434959:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434963:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757578.434968:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.434974:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305305611 00000100:00000001:1.0:1787757578.434979:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305306305 00000100:00000001:1.0:1787757578.435007:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305305611 00000100:00000001:1.0:1787757578.435013:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305306305 00000100:00000001:1.0:1787757578.435017:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757578.435021:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.435026:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757578.435035:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757578/real 1787757578] req@ffff8ffb45c31500 x1874588659662336/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757578.435067:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757578.435070:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757578.435076:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757578.435079:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757578.435082:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435084:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757578.435086:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757578.435092:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659662336/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757578.435102:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435105:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757578.435108:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435110:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757578.435111:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435113:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757578.435134:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757578.435136:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757578.435140:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757578.435142:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757578.435147:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757578.435152:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435155:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757578.435157:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757578.435159:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757578.435163:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757578.435164:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757578.435166:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757578.435170:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757578.435171:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757578.435174:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757578.435176:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435179:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757578.435180:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757578.435212:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757578.435216:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757578.435219:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757578.435221:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757578.435225:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757578.435228:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757578.435231:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757578.435233:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435237:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757578.435239:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757578.435242:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757578.435247:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757578.435250:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757578.435253:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757578.435256:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757578.435258:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757578.435261:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757578.435264:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435268:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757578.435276:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757578.435279:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757578.435283:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757578.435287:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757579.009300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757579.009312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757579.009315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757579.009323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757579.009328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757579.009334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757579.009338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757579.009342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757579.009346:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757579.201369:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757579.201383:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757579.201396:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757579.201400:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757579.201497:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757579.201503:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757579.201507:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757580.033377:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757580.033394:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757580.033400:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757580.033410:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757580.033418:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757580.033427:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757580.033433:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757580.033439:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757580.033445:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757580.225300:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757580.225309:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757580.225318:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757580.225321:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757580.225349:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757580.225352:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757580.225356:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757581.057281:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757581.057290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757581.057293:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757581.057299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757581.057303:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757581.057308:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757581.057312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757581.057315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757581.057318:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757581.249328:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757581.249337:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757581.249344:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757581.249348:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757581.249351:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757581.249355:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757581.249359:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757581.249361:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757581.249364:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757581.249367:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757581.249456:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757581.249462:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757581.249467:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757582.081332:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757582.081342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757582.081346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757582.081352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757582.081357:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757582.081361:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757582.081364:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757582.081368:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757582.081371:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757582.273282:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757582.273290:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757582.273298:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757582.273301:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757582.273391:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757582.273397:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757582.273401:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757583.105309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757583.105318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757583.105321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757583.105328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757583.105332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757583.105337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757583.105340:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757583.105344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757583.105347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757583.297341:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757583.297350:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757583.297360:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757583.297362:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757583.297445:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757583.297454:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757583.297461:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757583.553278:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757583.553292:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757583.553296:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757583.553301:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757583.553307:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757583.553318:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10638 00000100:00080000:3.0:1787757583.553334:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757583.553340:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757583.553343:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757583.553348:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757583.553352:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757583.553357:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757583.553360:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757583.553363:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757583.553364:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757583.553370:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757583.553373:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757583.553376:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757583.553381:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757583.553385:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.553388:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757583.553391:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757583.553409:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757583.553411:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.553416:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757583.553420:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757583.553426:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.553443:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757583.553456:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659662464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757583.553471:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757583.553477:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.553499:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757583.553504:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757583.553508:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757583.553510:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757583.553512:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757583.553515:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757583.553518:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10638 00000100:00080000:3.0:1787757583.553522:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757583.553523:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757583.553525:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757583.553528:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757583.553530:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757583.553533:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757583.553535:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757583.553538:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757583.553541:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757583.553544:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757583.553548:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757583.553550:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757583.553558:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757583.553561:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.553563:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757583.553570:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757583.553573:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.553580:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757583.553580:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.553598:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757583.553598:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757583.553608:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.553609:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757583.553616:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757583.553618:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757583.553622:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757583.553623:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757583.553630:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.553635:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757583.553641:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757583.553641:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.553656:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757583.553663:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659662592/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757583.553674:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659662464:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757583.553702:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757583.553704:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757583.553712:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.553714:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757583.553720:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.553721:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.553737:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757583.553738:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757583.553750:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757583.553752:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757583.553755:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757583.553756:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.553768:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757583.553776:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10638 00000100:00080000:3.0:1787757583.553782:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757583.553786:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757583.553787:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757583.553797:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757583.553798:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000020:00000001:3.0:1787757583.553819:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757583.553820:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787757583.553828:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757583.553833:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757583.553837:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757583.553840:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757583.553844:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757583.553848:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757583.553851:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757583.553852:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757583.553858:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757583.553875:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757583.553876:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757583.553883:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757583.553888:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757583.553908:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.553910:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757583.553915:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757583.553917:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.553920:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757583.553921:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757583.553923:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757583.553928:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.553931:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757583.553935:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.553940:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757583.553943:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.553948:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757583.553955:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659662720/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757583.553958:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305311425 00000100:00000001:3.0:1787757583.553976:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757583.553980:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757583.553980:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.554001:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554002:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305311425 00000100:00080000:3.0:1787757583.554015:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757583.554018:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757583.554030:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757583.554030:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757583.554047:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757583.554048:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757583.554055:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757583.554058:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757583.554071:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757583.554082:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10638 00000100:00100000:1.0:1787757583.554084:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757583/real 1787757583] req@ffff8ffb45c32a00 x1874588659662464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757583.554092:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757583.554104:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757583.554110:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757583.554123:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757583.554133:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000400:00000001:1.0:1787757583.554135:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:3.0:1787757583.554139:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787757583.554141:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000020:00000001:3.0:1787757583.554151:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757583.554155:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757583.554155:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757583.554169:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757583.554172:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757583.554180:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757583.554231:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757583.554237:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757583.554239:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757583.554244:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757583.554247:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.554249:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757583.554251:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757583.554254:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554256:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757583.554258:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554264:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757583.554267:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757583.554276:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.554279:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757583.554286:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757583.554288:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659662464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757583.554290:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.554300:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757583.554317:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659662848/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757583.554345:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757583.554348:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757583.554353:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757583.554354:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757583.554370:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554371:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554392:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757583.554395:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554399:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757583.554402:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757583.554408:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757583.554415:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757583.554419:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757583.554424:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659662464:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757583.554430:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554432:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757583.554434:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757583.554437:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757583.554443:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757583.554446:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757583.554449:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757583.554453:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757583.554455:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757583.554458:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757583.554460:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554462:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554470:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757583.554473:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554476:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757583.554478:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757583.554480:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757583.554482:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757583.554484:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.554487:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757583.554490:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757583.554492:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757583.554496:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662592:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757583.554500:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757583.554503:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757583.554505:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.554507:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757583.554511:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554519:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757583.554521:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757583.554526:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757583.554544:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757583.554550:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757583.554553:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757583.554555:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554556:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757583.554560:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554564:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305310734 00000100:00000001:1.0:1787757583.554568:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305311425 00000100:00000001:1.0:1787757583.554571:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305310734 00000100:00000001:1.0:1787757583.554573:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305311425 00000100:00000001:1.0:1787757583.554575:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757583.554577:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554579:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757583.554584:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757583/real 1787757583] req@ffff8ffb45c30700 x1874588659662592/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757583.554593:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757583.554595:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757583.554597:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757583.554599:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757583.554601:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554603:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757583.554605:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757583.554609:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659662592/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757583.554616:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554618:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757583.554621:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554623:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757583.554624:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554626:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757583.554640:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757583.554643:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757583.554647:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757583.554649:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757583.554652:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662592:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757583.554657:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757583.554658:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757583.554660:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.554662:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757583.554664:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757583.554666:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757583.554669:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757583.554672:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757583.554674:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757583.554676:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.554678:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757583.554681:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554686:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757583.554687:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787757583.554691:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757583.554701:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757583.554705:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757583.554707:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757583.554709:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554711:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757583.554713:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554717:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305310731 00000100:00000001:1.0:1787757583.554720:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305311425 00000100:00000001:1.0:1787757583.554723:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305310731 00000100:00000001:1.0:1787757583.554725:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305311425 00000100:00000001:1.0:1787757583.554728:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757583.554730:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554731:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757583.554736:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757583/real 1787757583] req@ffff8ffb45c30000 x1874588659662720/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757583.554744:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757583.554746:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757583.554748:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757583.554749:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757583.554751:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554753:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757583.554754:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757583.554758:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659662720/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757583.554766:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554768:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757583.554770:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554772:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757583.554773:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554775:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757583.554787:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757583.554789:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757583.554792:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757583.554794:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757583.554798:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757583.554801:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757583.554803:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757583.554805:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.554807:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757583.554809:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757583.554811:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757583.554815:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757583.554817:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757583.554820:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757583.554821:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757583.554824:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757583.554826:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554830:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757583.554832:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757583.554836:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757583.554845:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757583.554849:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757583.554851:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757583.554852:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554854:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757583.554857:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554860:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305310731 00000100:00000001:1.0:1787757583.554863:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305311425 00000100:00000001:1.0:1787757583.554865:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305310731 00000100:00000001:1.0:1787757583.554868:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305311425 00000100:00000001:1.0:1787757583.554870:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757583.554872:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.554874:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757583.554878:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757583/real 1787757583] req@ffff8ffb45c30e00 x1874588659662848/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757583.554885:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757583.554886:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757583.554888:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757583.554890:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757583.554891:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554917:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757583.554919:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757583.554924:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659662848/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757583.554948:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554951:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757583.554954:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554956:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757583.554957:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554959:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757583.554971:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757583.554974:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757583.554977:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757583.554980:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757583.554983:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659662848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757583.554986:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.554988:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757583.554990:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757583.554991:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757583.554993:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757583.554995:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757583.554997:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757583.554999:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757583.555000:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757583.555002:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757583.555004:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.555005:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757583.555007:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757583.555008:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757583.555011:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757583.555012:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757583.555013:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757583.555015:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757583.555016:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757583.555017:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757583.555018:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.555020:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757583.555021:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757583.555022:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757583.555023:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757583.555024:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757583.555025:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757583.555026:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757583.555028:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757583.555030:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757583.555031:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.555033:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757583.555037:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757583.555039:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757583.555041:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757583.555043:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757584.129280:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757584.129289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757584.129292:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757584.129298:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757584.129303:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757584.129308:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757584.129312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757584.129315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757584.129319:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757584.321308:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757584.321319:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757584.321325:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757584.321331:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757584.321339:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757584.321342:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757584.321346:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757584.321349:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757584.321356:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757584.321359:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757584.321397:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757584.321404:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757584.321411:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757585.153295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757585.153305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757585.153308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757585.153314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757585.153319:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757585.153324:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757585.153327:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757585.153331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757585.153335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757585.345298:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757585.345307:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757585.345317:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757585.345321:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757585.345393:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757585.345400:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757585.345405:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757586.177288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757586.177298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757586.177301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757586.177307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757586.177311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757586.177316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757586.177320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757586.177323:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757586.177327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757586.369308:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757586.369317:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757586.369324:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757586.369326:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757586.369391:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757586.369397:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757586.369402:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757587.201291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757587.201301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757587.201304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757587.201311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757587.201316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757587.201320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757587.201323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757587.201327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757587.201331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757587.393348:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757587.393357:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757587.393365:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757587.393367:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757587.393510:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757587.393519:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757587.393526:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757588.225315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757588.225329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757588.225332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757588.225342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757588.225347:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757588.225352:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757588.225356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757588.225360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757588.225364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757588.417314:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757588.417322:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757588.417330:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757588.417332:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757588.417360:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757588.417363:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757588.417367:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757588.673319:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757588.673335:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757588.673340:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757588.673346:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757588.673353:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757588.673364:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10643 00000100:00080000:3.0:1787757588.673382:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757588.673387:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757588.673390:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757588.673396:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757588.673401:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757588.673405:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757588.673409:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757588.673412:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757588.673414:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757588.673421:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757588.673425:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757588.673428:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757588.673433:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757588.673438:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673441:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757588.673444:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757588.673463:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757588.673466:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673471:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757588.673475:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757588.673482:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673501:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757588.673517:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659662976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757588.673533:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757588.673540:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673564:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757588.673569:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757588.673575:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757588.673577:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757588.673579:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757588.673583:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757588.673586:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10643 00000100:00080000:3.0:1787757588.673590:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757588.673592:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757588.673594:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757588.673597:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757588.673599:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757588.673602:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757588.673605:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757588.673607:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757588.673609:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757588.673613:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757588.673615:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757588.673630:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757588.673637:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757588.673640:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673642:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757588.673648:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757588.673653:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673656:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757588.673661:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757588.673665:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673669:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757588.673671:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757588.673674:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673680:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757588.673688:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659663104/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757588.673692:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757588.673707:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757588.673708:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757588.673726:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757588.673727:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.673741:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757588.673744:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.673747:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757588.673755:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757588.673759:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757588.673765:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787757588.673768:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757588.673772:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757588.673780:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757588.673782:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673786:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757588.673796:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10643 00000100:00080000:3.0:1787757588.673822:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757588.673826:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757588.673832:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757588.673836:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659662976:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757588.673837:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757588.673840:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757588.673847:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757588.673856:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757588.673869:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757588.673870:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757588.673872:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757588.673883:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757588.673884:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757588.673888:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757588.673893:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757588.673893:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757588.673902:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757588.673907:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757588.673908:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673919:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757588.673924:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757588.673929:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757588.673930:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673936:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757588.673949:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757588.673951:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.673953:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757588.673956:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757588.673959:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.673966:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757588.673967:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.673975:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00100000:3.0:1787757588.673987:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659663232/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757588.673989:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757588.674004:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757588.674008:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.674013:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757588.674017:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757588.674021:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757588.674023:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757588.674025:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757588.674029:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757588.674034:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10643 00000100:00080000:3.0:1787757588.674038:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757588.674041:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757588.674043:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757588.674046:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757588.674051:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000800:00000100:1.0:1787757588.674057:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787757588.674059:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757588.674069:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757588.674073:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757588.674076:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757588.674077:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757588.674087:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757588.674090:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757588.674093:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674097:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757588.674099:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757588.674107:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757588.674111:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757588.674117:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674120:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.674122:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757588.674132:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757588.674133:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305316545 00000008:00000001:3.0:1787757588.674155:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.674161:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674164:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757588.674169:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757588.674174:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.674178:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757588.674179:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305316545 02000000:00000001:3.0:1787757588.674236:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757588.674242:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674251:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757588.674253:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674267:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:3.0:1787757588.674271:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659663360/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757588.674281:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674300:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757588.674303:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757588.674308:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757588.674312:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757588.674319:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757588/real 1787757588] req@ffff8ffb45c31c00 x1874588659662976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757588.674337:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757588.674339:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757588.674345:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757588.674351:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757588.674354:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674358:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757588.674361:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757588.674367:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659662976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757588.674378:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674382:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757588.674386:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674389:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757588.674390:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674394:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757588.674396:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757588.674400:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757588.674405:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757588.674408:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757588.674413:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659662976:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757588.674419:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674422:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757588.674424:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757588.674427:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757588.674434:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757588.674437:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757588.674440:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757588.674445:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757588.674448:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757588.674451:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757588.674453:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674456:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674465:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757588.674467:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674471:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757588.674473:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757588.674475:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757588.674478:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757588.674480:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757588.674483:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757588.674486:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757588.674489:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757588.674493:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663104:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757588.674497:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757588.674501:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757588.674503:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757588.674505:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757588.674509:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674516:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757588.674518:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757588.674523:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757588.674540:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757588.674546:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757588.674549:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757588.674551:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674553:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757588.674557:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674561:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305315854 00000100:00000001:1.0:1787757588.674566:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305316545 00000100:00000001:1.0:1787757588.674569:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305315854 00000100:00000001:1.0:1787757588.674572:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305316545 00000100:00000001:1.0:1787757588.674575:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757588.674577:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674580:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757588.674586:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757588/real 1787757588] req@ffff8ffb45c33b80 x1874588659663104/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757588.674596:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757588.674598:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757588.674601:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757588.674603:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757588.674605:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674607:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757588.674609:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757588.674614:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659663104/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757588.674622:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674625:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757588.674627:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674629:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757588.674631:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674633:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757588.674648:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757588.674651:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757588.674656:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757588.674658:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757588.674663:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663104:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757588.674667:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757588.674669:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757588.674671:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757588.674674:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757588.674676:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757588.674678:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757588.674682:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757588.674689:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757588.674691:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757588.674694:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757588.674696:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757588.674699:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674705:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757588.674707:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757588.674711:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757588.674722:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757588.674727:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757588.674730:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757588.674731:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674733:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757588.674736:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674740:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305315851 00000100:00000001:1.0:1787757588.674744:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305316545 00000100:00000001:1.0:1787757588.674746:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305315851 00000100:00000001:1.0:1787757588.674749:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305316545 00000100:00000001:1.0:1787757588.674752:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757588.674754:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674756:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757588.674761:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757588/real 1787757588] req@ffff8ffb45c30a80 x1874588659663232/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757588.674769:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757588.674771:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757588.674773:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757588.674775:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757588.674776:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674779:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757588.674781:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757588.674789:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659663232/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757588.674838:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674843:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757588.674848:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674851:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757588.674854:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.674858:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757588.674877:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757588.674883:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757588.674891:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757588.674895:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757588.674902:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757588.674910:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757588.674913:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757588.674916:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757588.674920:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757588.674923:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757588.674925:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757588.674930:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663360:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757588.674934:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757588.674937:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757588.674939:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757588.674941:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757588.674946:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674954:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757588.674956:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757588.674961:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757588.674978:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757588.674984:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757588.674987:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757588.674989:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.674992:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757588.674996:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.675000:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305315851 00000100:00000001:1.0:1787757588.675003:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305316545 00000100:00000001:1.0:1787757588.675006:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305315851 00000100:00000001:1.0:1787757588.675009:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305316545 00000100:00000001:1.0:1787757588.675012:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757588.675014:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.675016:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757588.675021:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757588/real 1787757588] req@ffff8ffb45c30e00 x1874588659663360/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757588.675031:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757588.675032:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757588.675035:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757588.675037:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757588.675039:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675041:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757588.675043:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757588.675047:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659663360/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757588.675056:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675059:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757588.675062:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675064:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757588.675066:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675068:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757588.675081:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757588.675084:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757588.675087:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757588.675090:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757588.675094:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663360:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757588.675099:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675101:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757588.675103:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757588.675105:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757588.675108:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757588.675110:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757588.675112:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757588.675114:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757588.675116:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757588.675119:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757588.675120:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675122:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757588.675124:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757588.675126:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757588.675127:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757588.675129:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757588.675130:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757588.675132:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757588.675133:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757588.675135:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757588.675137:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675139:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757588.675140:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757588.675141:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757588.675144:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757588.675145:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757588.675146:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757588.675148:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757588.675150:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757588.675151:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757588.675153:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675155:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757588.675160:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757588.675161:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757588.675164:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757588.675166:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757589.249358:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757589.249370:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757589.249373:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757589.249382:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757589.249388:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757589.249397:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757589.249402:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757589.249408:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757589.249414:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757589.441395:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757589.441409:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757589.441419:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757589.441422:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757589.441501:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757589.441508:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757589.441514:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757590.273286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757590.273295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757590.273298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757590.273305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757590.273309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757590.273314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757590.273317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757590.273321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757590.273325:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757590.465375:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757590.465382:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757590.465389:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757590.465391:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757590.465460:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757590.465466:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757590.465471:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757591.297228:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757591.297234:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757591.297235:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757591.297239:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757591.297242:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757591.297245:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757591.297247:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757591.297249:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757591.297251:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757591.489281:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757591.489289:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757591.489298:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757591.489303:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757591.489307:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757591.489311:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757591.489314:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757591.489317:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757591.489320:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757591.489323:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757591.489383:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757591.489390:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757591.489395:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757592.321278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757592.321287:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757592.321290:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757592.321297:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757592.321302:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757592.321307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757592.321310:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757592.321313:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757592.321317:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757592.513341:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757592.513352:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757592.513360:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757592.513362:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757592.513395:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757592.513400:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757592.513404:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757593.345346:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757593.345358:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757593.345362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757593.345369:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757593.345375:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757593.345381:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757593.345384:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757593.345388:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757593.345392:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757593.537322:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757593.537332:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757593.537342:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757593.537345:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757593.537415:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757593.537421:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757593.537426:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757593.793297:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757593.793310:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757593.793314:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757593.793319:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757593.793325:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757593.793334:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10648 00000100:00080000:3.0:1787757593.793347:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757593.793352:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757593.793354:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757593.793360:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757593.793363:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757593.793366:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757593.793370:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757593.793373:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757593.793390:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757593.793396:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757593.793399:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757593.793403:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757593.793408:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757593.793412:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793415:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757593.793418:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757593.793431:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757593.793433:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793437:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757593.793441:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757593.793446:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793461:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757593.793473:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659663488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757593.793487:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757593.793493:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793512:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757593.793517:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757593.793521:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757593.793523:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757593.793526:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757593.793529:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757593.793532:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10648 00000100:00080000:3.0:1787757593.793537:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757593.793540:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757593.793542:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757593.793545:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757593.793547:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757593.793550:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757593.793553:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757593.793556:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757593.793557:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757593.793561:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757593.793564:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757593.793566:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757593.793572:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757593.793574:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793576:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757593.793582:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757593.793586:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793589:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757593.793595:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757593.793597:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793600:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757593.793602:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757593.793604:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793610:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757593.793617:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659663616/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757593.793620:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757593.793630:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757593.793634:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757593.793637:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757593.793640:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.793645:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757593.793648:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.793650:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757593.793657:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757593.793661:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757593.793664:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787757593.793668:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757593.793669:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757593.793675:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757593.793679:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793682:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757593.793692:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10648 00000100:00100000:1.0:1787757593.793693:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659663488:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757593.793697:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757593.793706:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757593.793710:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757593.793711:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757593.793741:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757593.793746:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757593.793760:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787757593.793761:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787757593.793767:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757593.793771:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787757593.793774:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757593.793784:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787757593.793786:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793789:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757593.793803:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757593.793809:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.793813:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757593.793814:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757593.793823:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 02000000:00000001:3.0:1787757593.793827:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757593.793838:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757593.793839:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793848:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757593.793854:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757593.793860:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793864:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757593.793873:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757593.793875:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793877:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757593.793880:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757593.793884:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793895:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757593.793903:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659663744/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757593.793914:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757593.793917:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.793922:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757593.793925:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757593.793929:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757593.793932:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757593.793935:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757593.793939:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757593.793941:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757593.793945:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10648 00000100:00000001:1.0:1787757593.793952:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305321665 00000100:00080000:3.0:1787757593.793955:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757593.793963:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757593.793966:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757593.793970:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757593.793973:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787757593.793974:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757593.793980:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757593.793996:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757593.793999:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305321665 00000100:00000001:3.0:1787757593.794000:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757593.794005:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757593.794014:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757593.794017:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794020:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757593.794021:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757593.794029:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757593.794034:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794037:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757593.794038:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757593.794044:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757593.794051:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757593.794052:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.794064:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794068:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757593.794072:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757593.794077:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794079:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757593.794081:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757593.794088:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757593.794091:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794098:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757593.794099:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794108:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757593.794117:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757593.794119:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757593.794123:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:3.0:1787757593.794129:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659663872/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757593.794130:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757593.794144:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757593.794152:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757593.794153:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757593.794165:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757593.794167:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757593.794172:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757593.794178:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757593.794245:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663616:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757593.794253:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757593.794261:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757593.794264:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757593.794268:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757593.794275:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794288:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757593.794294:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757593.794303:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757593.794336:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794345:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305320974 00000100:00000001:1.0:1787757593.794353:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305321665 00000100:00000001:1.0:1787757593.794359:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305320974 00000100:00000001:1.0:1787757593.794364:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305321665 00000100:00000001:1.0:1787757593.794368:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757593.794372:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794377:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757593.794380:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757593.794383:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757593.794389:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757593.794394:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757593.794398:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757593.794405:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757593.794411:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757593.794416:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757593.794419:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757593.794423:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757593.794429:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794442:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757593.794446:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787757593.794455:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757593.794481:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794490:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305320971 00000100:00000001:1.0:1787757593.794496:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305321665 00000100:00000001:1.0:1787757593.794501:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305320971 00000100:00000001:1.0:1787757593.794507:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305321665 00000100:00000001:1.0:1787757593.794511:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757593.794516:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794520:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794524:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757593.794533:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757593.794536:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757593.794541:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757593.794545:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757593.794548:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757593.794551:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757593.794554:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757593.794559:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757593.794563:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757593.794568:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757593.794577:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663872:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757593.794585:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757593.794590:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757593.794592:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757593.794595:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757593.794601:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794613:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757593.794616:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757593.794623:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757593.794642:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794647:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305320971 00000100:00000001:1.0:1787757593.794654:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305321665 00000100:00000001:1.0:1787757593.794672:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305320971 00000100:00000001:1.0:1787757593.794678:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305321665 00000100:00000001:1.0:1787757593.794683:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757593.794687:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794694:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794699:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757593.794708:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757593.794752:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757593.794760:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757593.794763:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757593.794769:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757593.794774:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757594.369290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757594.369300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757594.369303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757594.369309:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757594.369314:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757594.369318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757594.369321:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757594.369325:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757594.369328:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757594.561309:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757594.561320:0:178197:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757594.561323:0:178197:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757594.561328:0:178197:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757594.561339:0:178197:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757594.561344:0:178197:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757594.561349:0:178197:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757594.561356:0:178197:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757594.561365:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757594.561370:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757594.561454:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757594.561461:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757594.561466:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757595.393310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757595.393320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757595.393322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757595.393329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757595.393334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757595.393338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757595.393342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757595.393345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757595.393349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757595.585308:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757595.585334:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757595.585345:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757595.585347:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757595.585501:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757595.585507:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757595.585512:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757596.417307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757596.417315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757596.417317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757596.417322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757596.417326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757596.417330:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757596.417332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757596.417335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757596.417338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757596.609416:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757596.609426:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757596.609434:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757596.609436:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757596.609469:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757596.609472:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757596.609476:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787757596.865839:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787757596.865934:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787757596.865968:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757596.865996:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757596.866037:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757596.866041:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757596.866044:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757596.866062:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757596.866072:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757596.866078:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757596.866082:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757596.866086:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757596.866095:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757596.866104:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757596.866110:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757596.866114:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757596.866117:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757596.866129:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757596.866135:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757596.866139:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757596.866147:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757596.866150:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757596.866156:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757596.866158:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757596.866161:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:1.0:1787757596.866254:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757593/real 1787757596] req@ffff8ffb45c31500 x1874588659663872/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757596.866286:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757596.866292:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757596.866316:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757596.866322:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757596.866327:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866333:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757596.866337:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757596.866348:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659663872/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757596.866363:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866370:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757596.866376:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866380:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757596.866382:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866388:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757596.866420:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757596.866428:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757596.866436:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757596.866441:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757596.866450:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663872:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757596.866459:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757596.866468:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757593/real 1787757596] req@ffff8ffb45c30700 x1874588659663616/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757596.866484:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757596.866487:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757596.866493:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757596.866497:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757596.866499:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866503:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757596.866505:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757596.866512:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659663616/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757596.866526:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866530:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757596.866534:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866538:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757596.866540:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866543:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757596.866601:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757596.866607:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757596.866615:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757596.866620:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757596.866627:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663616:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757596.866636:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757596.866644:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757593/real 1787757596] req@ffff8ffb45c32a00 x1874588659663744/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757596.866689:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757596.866692:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757596.866698:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757596.866702:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757596.866705:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866711:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757596.866714:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757596.866722:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659663744/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757596.866736:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866741:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757596.866746:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866749:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757596.866751:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866755:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757596.866781:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757596.866787:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757596.866794:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757596.866798:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757596.866805:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659663744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757596.866813:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757596.866821:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757593/real 1787757596] req@ffff8ffb45c30000 x1874588659663488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757596.866836:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757596.866839:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757596.866846:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757596.866849:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757596.866852:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866856:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757596.866858:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757596.866866:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659663488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757596.866881:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866886:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757596.866890:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866894:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757596.866897:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866900:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757596.866903:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757596.866908:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757596.866916:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757596.866920:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757596.866927:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659663488:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757596.866938:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.866943:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757596.866946:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757596.866952:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757596.866964:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757596.866970:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757596.866975:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757596.866985:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757596.866989:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757596.866995:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757596.866998:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.867002:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757596.867004:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757596.867007:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757596.867013:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757596.867015:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757596.867018:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757596.867022:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757596.867025:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757596.867028:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757596.867031:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.867035:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757596.867037:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757596.867040:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757596.867056:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757596.867059:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757596.867062:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757596.867066:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757596.867068:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757596.867072:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757596.867076:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.867080:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757596.867083:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757596.867085:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757596.867091:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757596.867093:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757596.867095:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757596.867099:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757596.867101:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757596.867104:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757596.867106:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.867111:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757596.867126:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757596.867130:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757596.867135:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757596.867139:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757597.441242:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757597.441250:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757597.441252:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757597.441258:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757597.441268:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757597.441271:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757597.441274:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757597.441277:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757597.441279:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757597.633341:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757597.633351:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757597.633359:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757597.633361:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757597.633477:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757597.633483:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757597.633488:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757598.465314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757598.465324:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757598.465327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757598.465333:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757598.465338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757598.465342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757598.465346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757598.465349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757598.465352:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757598.657318:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757598.657326:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757598.657333:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757598.657335:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757598.657450:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757598.657460:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757598.657467:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757598.913319:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757598.913332:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757598.913336:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757598.913340:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757598.913347:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757598.913357:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10653 00000100:00080000:3.0:1787757598.913371:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757598.913377:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757598.913379:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757598.913385:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757598.913388:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757598.913392:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757598.913414:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757598.913418:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757598.913421:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757598.913437:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757598.913443:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757598.913449:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757598.913455:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757598.913461:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.913465:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757598.913470:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757598.913485:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757598.913489:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.913495:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757598.913501:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757598.913508:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.913527:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757598.913543:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659664000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757598.913595:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757598.913603:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.913659:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757598.913667:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757598.913675:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757598.913679:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757598.913683:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757598.913690:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757598.913698:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10653 00000100:00080000:3.0:1787757598.913705:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757598.913711:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757598.913711:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757598.913718:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757598.913722:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:3.0:1787757598.913728:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757598.913731:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:3.0:1787757598.913733:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757598.913742:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000020:00000001:3.0:1787757598.913742:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757598.913751:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757598.913752:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757598.913764:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757598.913766:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757598.913768:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757598.913777:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757598.913778:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757598.913788:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.913790:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757598.913799:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:1.0:1787757598.913809:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659664000:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757598.913815:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757598.913826:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757598.913827:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.913837:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757598.913839:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:3.0:1787757598.913846:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757598.913847:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.913857:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00000001:3.0:1787757598.913862:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.913869:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.913871:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757598.913889:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757598.913892:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.913899:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757598.913902:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757598.913905:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757598.913908:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.913914:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757598.913926:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757598.913926:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757598.913970:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659664128/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757598.913990:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757598.913995:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.914004:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757598.914011:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757598.914014:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757598.914025:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757598.914030:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757598.914034:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757598.914037:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757598.914041:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757598.914051:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757598.914054:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10653 00000100:00000001:1.0:1787757598.914061:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757598.914069:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757598.914070:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757598.914081:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757598.914085:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757598.914088:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757598.914092:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757598.914100:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757598.914101:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305326785 00000020:00000001:3.0:1787757598.914105:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757598.914116:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757598.914121:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757598.914122:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757598.914129:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757598.914138:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305326785 00000100:00080000:3.0:1787757598.914143:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757598.914154:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757598.914155:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757598.914167:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757598.914169:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757598.914174:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757598.914180:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757598.914229:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.914235:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757598.914240:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757598.914248:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.914251:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914293:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757598.914317:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757598/real 1787757598] req@ffff8ffb45c33800 x1874588659664000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757598.914342:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757598.914346:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757598.914353:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757598.914359:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757598.914362:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914367:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757598.914370:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757598.914380:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659664000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757598.914396:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914403:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757598.914410:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914414:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757598.914418:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914423:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757598.914427:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757598.914433:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757598.914442:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757598.914448:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757598.914456:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659664000:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757598.914465:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914471:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757598.914474:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757598.914479:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757598.914486:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757598.914490:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757598.914492:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757598.914497:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757598.914500:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757598.914504:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757598.914506:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914509:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914518:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757598.914520:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914523:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757598.914527:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757598.914530:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757598.914533:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757598.914536:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.914540:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757598.914544:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757598.914549:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757598.914556:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664128:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757598.914563:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757598.914568:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757598.914571:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.914575:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757598.914581:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914595:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757598.914598:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757598.914605:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757598.914668:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757598.914679:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757598.914685:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757598.914688:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914691:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757598.914698:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914705:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305326094 00000100:00000001:1.0:1787757598.914711:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305326785 00000100:00000001:1.0:1787757598.914717:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305326094 00000100:00000001:1.0:1787757598.914721:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305326785 00000100:00000001:1.0:1787757598.914725:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757598.914729:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914733:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757598.914742:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757598/real 1787757598] req@ffff8ffb45c31180 x1874588659664128/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757598.914761:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757598.914763:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757598.914769:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757598.914773:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757598.914804:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914808:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757598.914811:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757598.914821:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659664128/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757598.914835:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757598.914839:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757598.914843:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914848:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757598.914853:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757598.914859:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757598.914862:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757598.914869:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757598.914872:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914877:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.914885:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757598.914893:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757598.914914:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659664256/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757598.914924:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757598.914939:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757598.914942:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757598.914952:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757598.914958:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757598.914965:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.914968:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00080000:3.0:1787757598.914976:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757598.914983:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664128:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1787757598.914988:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757598.914999:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757598.915003:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757598.915007:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757598.915011:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757598.915017:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757598.915020:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757598.915026:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757598.915033:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00080000:3.0:1787757598.915033:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10653 00000020:00000001:1.0:1787757598.915039:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757598.915046:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757598.915053:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00080000:3.0:1787757598.915053:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757598.915059:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757598.915063:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757598.915067:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757598.915072:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757598.915074:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915086:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757598.915086:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757598.915095:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757598.915100:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000020:00000001:3.0:1787757598.915103:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757598.915108:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757598.915114:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757598.915117:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757598.915122:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757598.915126:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757598.915129:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757598.915133:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757598.915137:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757598.915145:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.915151:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757598.915152:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757598.915155:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757598.915167:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.915169:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757598.915178:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664256:192.168.201.118@tcp:8:kworker.0 02000000:00000001:3.0:1787757598.915219:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757598.915229:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757598.915241:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757598.915242:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.915250:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757598.915252:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757598.915259:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757598.915262:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757598.915268:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00000001:3.0:1787757598.915273:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.915280:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.915281:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757598.915301:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757598.915304:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915306:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757598.915311:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757598.915314:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 02000000:00000001:3.0:1787757598.915319:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757598.915326:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757598.915327:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.915345:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757598.915358:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659664384/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757598.915368:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757598.915381:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757598.915388:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757598.915394:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757598.915402:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757598.915409:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915415:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757598.915422:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915427:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915439:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305326092 00000100:00000001:1.0:1787757598.915446:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305326785 00000100:00000001:1.0:1787757598.915451:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305326092 00000100:00000001:1.0:1787757598.915458:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305326785 00000100:00000001:1.0:1787757598.915463:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757598.915468:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915473:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757598.915485:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757598/real 1787757598] req@ffff8ffb45c33480 x1874588659664256/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757598.915512:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757598.915514:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757598.915521:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757598.915525:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757598.915528:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915532:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757598.915535:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757598.915543:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659664256/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757598.915558:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915562:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757598.915567:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915569:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757598.915571:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915573:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757598.915589:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757598.915593:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757598.915597:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757598.915600:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757598.915605:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757598.915610:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915613:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757598.915615:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757598.915617:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757598.915645:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757598.915648:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757598.915651:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757598.915656:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757598.915658:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757598.915662:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757598.915664:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915668:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915675:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757598.915678:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915682:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757598.915685:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757598.915688:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757598.915691:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757598.915694:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.915699:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757598.915703:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757598.915707:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757598.915715:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664384:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757598.915732:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757598.915736:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757598.915739:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757598.915743:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757598.915749:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915761:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757598.915763:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757598.915768:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757598.915788:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757598.915795:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757598.915799:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757598.915801:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915803:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757598.915807:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915811:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305326092 00000100:00000001:1.0:1787757598.915814:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305326785 00000100:00000001:1.0:1787757598.915817:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305326092 00000100:00000001:1.0:1787757598.915820:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305326785 00000100:00000001:1.0:1787757598.915824:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757598.915826:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915829:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757598.915834:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757598/real 1787757598] req@ffff8ffb45c31180 x1874588659664384/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757598.915845:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757598.915847:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757598.915850:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757598.915852:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757598.915854:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915856:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757598.915858:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757598.915862:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659664384/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757598.915871:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915874:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757598.915877:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915879:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757598.915880:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915883:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757598.915898:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757598.915901:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757598.915905:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757598.915908:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757598.915913:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664384:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757598.915917:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915919:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757598.915921:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757598.915923:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757598.915926:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757598.915927:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757598.915930:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757598.915932:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757598.915934:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757598.915936:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757598.915938:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915940:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757598.915945:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757598.915946:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757598.915949:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757598.915951:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757599.489319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757599.489331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757599.489336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757599.489345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757599.489352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757599.489357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757599.489361:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757599.489363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757599.489367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757599.681400:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757599.681413:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757599.681427:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757599.681431:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757599.681512:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757599.681518:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757599.681523:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757600.513306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757600.513323:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757600.513328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757600.513336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757600.513342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757600.513348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757600.513352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757600.513356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757600.513361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757600.705373:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757600.705386:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757600.705399:0:178197:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757600.705402:0:178197:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757600.705448:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757600.705455:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757600.705460:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757601.537349:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757601.537363:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757601.537366:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757601.537373:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757601.537378:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757601.537383:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757601.537387:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757601.537390:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757601.537394:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757601.729431:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757601.729442:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757601.729450:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757601.729457:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757601.729461:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757601.729466:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757601.729470:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757601.729474:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757601.729477:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757601.729480:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757601.729596:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757601.729605:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757601.729611:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757602.561343:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757602.561354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757602.561359:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757602.561367:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757602.561387:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757602.561393:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757602.561397:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757602.561400:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757602.561403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757602.753347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757602.753356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757602.753363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757602.753364:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757602.753433:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757602.753439:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757602.753444:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757603.585395:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757603.585408:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757603.585412:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757603.585420:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757603.585426:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757603.585433:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757603.585438:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757603.585444:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757603.585450:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757603.777407:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757603.777417:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757603.777426:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757603.777428:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757603.777506:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757603.777514:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757603.777520:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757604.033279:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757604.033292:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757604.033296:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757604.033300:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757604.033307:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757604.033317:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10658 00000100:00080000:3.0:1787757604.033331:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757604.033335:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757604.033337:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757604.033343:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757604.033346:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757604.033350:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757604.033353:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757604.033356:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757604.033358:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757604.033365:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757604.033368:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757604.033371:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757604.033375:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757604.033379:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033382:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757604.033385:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757604.033399:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757604.033401:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033405:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757604.033408:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757604.033414:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033428:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757604.033441:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659664512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757604.033455:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757604.033461:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033492:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757604.033498:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757604.033502:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757604.033504:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757604.033507:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757604.033512:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757604.033515:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10658 00000100:00080000:3.0:1787757604.033520:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757604.033523:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757604.033525:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757604.033528:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757604.033529:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757604.033557:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757604.033561:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757604.033565:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757604.033569:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757604.033577:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757604.033582:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757604.033586:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757604.033595:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757604.033600:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033605:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757604.033613:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757604.033621:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033626:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757604.033635:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757604.033638:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757604.033640:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033649:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757604.033650:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757604.033653:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757604.033659:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757604.033662:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.033667:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757604.033675:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033677:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757604.033687:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:3.0:1787757604.033690:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659664640/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757604.033696:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757604.033709:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033713:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757604.033720:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757604.033726:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659664512:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757604.033727:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757604.033736:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757604.033740:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757604.033743:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757604.033748:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:1.0:1787757604.033749:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787757604.033751:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757604.033755:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033758:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757604.033761:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757604.033767:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10658 00000100:00080000:3.0:1787757604.033774:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757604.033777:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757604.033779:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757604.033782:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757604.033784:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757604.033788:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787757604.033789:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757604.033794:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757604.033800:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757604.033802:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757604.033807:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757604.033810:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757604.033813:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757604.033816:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757604.033818:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757604.033824:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 02000000:00000001:3.0:1787757604.033826:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033836:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757604.033838:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00100000:3.0:1787757604.033841:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757604.033847:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033849:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757604.033856:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757604.033858:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033861:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757604.033862:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757604.033865:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033871:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757604.033879:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659664768/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757604.033897:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757604.033911:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757604.033915:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.033921:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.033922:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757604.033925:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757604.033931:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757604.033932:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757604.033939:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757604.033940:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757604.033942:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757604.033945:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757604.033949:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757604.033954:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757604.033961:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10658 00000100:00000001:1.0:1787757604.033964:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305331905 00000100:00080000:3.0:1787757604.033975:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757604.033981:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757604.033983:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757604.033991:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757604.033995:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305331905 00000020:00000001:3.0:1787757604.033998:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757604.034004:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:3.0:1787757604.034005:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757604.034014:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757604.034015:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787757604.034026:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757604.034027:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034034:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757604.034037:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757604.034045:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757604.034047:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757604/real 1787757604] req@ffff8ffb45c33800 x1874588659664512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757604.034061:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757604.034071:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757604.034080:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000400:00000001:1.0:1787757604.034082:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787757604.034085:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:1.0:1787757604.034088:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757604.034094:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034098:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757604.034099:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757604.034107:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757604.034108:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000008:00000001:3.0:1787757604.034115:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034116:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757604.034121:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034124:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757604.034129:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757604.034131:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757604.034136:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757604.034139:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659664512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757604.034142:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757604.034147:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757604.034152:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034162:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757604.034163:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034168:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757604.034176:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757604.034178:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659664896/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757604.034180:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757604.034234:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757604.034238:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757604.034241:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034248:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034255:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757604.034258:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757604.034263:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757604.034270:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757604.034276:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757604.034285:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659664512:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757604.034293:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034296:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757604.034299:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757604.034302:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757604.034309:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757604.034311:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757604.034314:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757604.034321:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757604.034323:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757604.034326:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757604.034327:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034330:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034339:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757604.034342:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034344:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757604.034346:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757604.034349:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757604.034351:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757604.034352:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757604.034356:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757604.034358:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757604.034361:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757604.034365:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664640:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757604.034369:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757604.034371:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757604.034373:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757604.034375:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757604.034379:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034387:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757604.034388:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787757604.034394:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757604.034410:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757604.034416:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757604.034419:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757604.034421:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034423:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757604.034426:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034432:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305331214 00000100:00000001:1.0:1787757604.034435:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305331905 00000100:00000001:1.0:1787757604.034439:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305331214 00000100:00000001:1.0:1787757604.034442:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305331905 00000100:00000001:1.0:1787757604.034444:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757604.034448:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034450:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757604.034456:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757604/real 1787757604] req@ffff8ffb45c30000 x1874588659664640/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757604.034467:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757604.034469:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757604.034472:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757604.034474:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757604.034475:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034477:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757604.034479:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757604.034484:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659664640/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757604.034493:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034495:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757604.034498:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034500:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757604.034502:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034504:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757604.034518:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757604.034521:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757604.034524:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757604.034527:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757604.034559:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664640:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757604.034566:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757604.034568:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757604.034569:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757604.034572:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757604.034574:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757604.034577:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757604.034581:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757604.034586:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757604.034588:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757604.034590:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757604.034593:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757604.034597:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034606:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757604.034609:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787757604.034616:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757604.034639:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757604.034647:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757604.034651:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757604.034654:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034658:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757604.034664:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034670:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305331211 00000100:00000001:1.0:1787757604.034675:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305331905 00000100:00000001:1.0:1787757604.034680:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305331211 00000100:00000001:1.0:1787757604.034684:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305331905 00000100:00000001:1.0:1787757604.034689:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757604.034693:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034697:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757604.034705:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757604/real 1787757604] req@ffff8ffb45c32a00 x1874588659664768/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757604.034723:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757604.034725:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757604.034729:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757604.034732:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757604.034734:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034736:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757604.034738:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757604.034743:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659664768/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757604.034761:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034764:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757604.034767:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034769:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757604.034770:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034772:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757604.034786:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757604.034788:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757604.034792:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757604.034795:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757604.034799:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757604.034804:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757604.034806:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757604.034808:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757604.034810:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757604.034813:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757604.034815:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757604.034819:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757604.034823:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757604.034825:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757604.034827:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757604.034829:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757604.034833:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034839:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757604.034841:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757604.034846:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757604.034861:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757604.034867:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757604.034869:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757604.034871:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034874:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757604.034877:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034881:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305331211 00000100:00000001:1.0:1787757604.034884:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305331905 00000100:00000001:1.0:1787757604.034887:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305331211 00000100:00000001:1.0:1787757604.034890:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305331905 00000100:00000001:1.0:1787757604.034892:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757604.034895:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.034897:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757604.034901:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757604/real 1787757604] req@ffff8ffb45c30700 x1874588659664896/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757604.034910:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757604.034911:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757604.034914:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757604.034916:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757604.034918:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034920:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757604.034922:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757604.034926:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659664896/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757604.034934:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034937:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757604.034939:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034941:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757604.034943:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034945:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757604.034956:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757604.034959:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757604.034963:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757604.034966:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757604.034970:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659664896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757604.034975:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034977:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757604.034979:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757604.034981:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757604.034984:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757604.034986:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757604.034988:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757604.034991:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757604.034993:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757604.034995:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757604.034997:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.034999:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757604.035000:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757604.035002:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757604.035005:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757604.035007:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757604.035008:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757604.035010:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757604.035011:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757604.035013:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757604.035015:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.035017:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757604.035018:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757604.035019:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757604.035021:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757604.035022:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757604.035024:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757604.035025:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757604.035027:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757604.035028:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757604.035030:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.035032:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757604.035036:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757604.035038:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757604.035040:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757604.035042:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757604.609326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757604.609335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757604.609337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757604.609345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757604.609349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757604.609353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757604.609356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757604.609360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757604.609363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757604.801345:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757604.801356:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757604.801360:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757604.801364:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757604.801371:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757604.801374:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757604.801378:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757604.801383:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757604.801391:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757604.801394:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757604.801426:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757604.801430:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757604.801434:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757605.633383:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757605.633398:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757605.633403:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757605.633413:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757605.633421:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757605.633429:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757605.633435:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757605.633441:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757605.633447:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757605.825362:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757605.825373:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757605.825380:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757605.825382:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757605.825566:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757605.825575:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757605.825581:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757606.657347:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757606.657360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757606.657363:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757606.657371:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757606.657376:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757606.657383:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757606.657387:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757606.657391:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757606.657395:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757606.849352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757606.849367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757606.849380:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757606.849385:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757606.849471:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757606.849502:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757606.849511:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757607.681368:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757607.681381:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757607.681384:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757607.681393:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757607.681400:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757607.681407:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757607.681413:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757607.681420:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757607.681426:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757607.873366:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757607.873374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757607.873381:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757607.873383:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757607.873494:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757607.873500:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757607.873506:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757608.705320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757608.705333:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757608.705336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757608.705344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757608.705349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757608.705355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757608.705359:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757608.705363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757608.705367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757608.897347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757608.897356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757608.897366:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757608.897370:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757608.897425:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757608.897434:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757608.897442:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757609.153294:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757609.153310:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757609.153314:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757609.153320:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757609.153326:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757609.153336:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10663 00000100:00080000:2.0:1787757609.153352:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757609.153356:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757609.153359:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757609.153365:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757609.153369:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757609.153374:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757609.153385:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757609.153388:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757609.153391:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757609.153398:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757609.153401:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757609.153405:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757609.153411:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757609.153417:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153421:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757609.153426:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757609.153467:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757609.153471:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153478:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757609.153483:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757609.153491:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153515:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757609.153535:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659665024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757609.153553:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757609.153560:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153585:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757609.153589:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757609.153594:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757609.153596:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757609.153599:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757609.153603:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757609.153608:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10663 00000100:00080000:2.0:1787757609.153614:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757609.153617:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757609.153620:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757609.153624:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757609.153627:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757609.153629:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:2.0:1787757609.153631:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757609.153638:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757609.153639:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757609.153645:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757609.153650:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757609.153653:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757609.153655:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757609.153657:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.153667:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:2.0:1787757609.153668:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757609.153677:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.153678:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757609.153688:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757609.153689:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757609.153701:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787757609.153703:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659665024:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757609.153713:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.153716:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757609.153718:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757609.153724:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:2.0:1787757609.153725:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757609.153727:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.153743:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00000001:2.0:1787757609.153743:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153751:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.153757:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757609.153759:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757609.153764:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153769:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757609.153771:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757609.153776:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.153777:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757609.153780:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:1.0:1787757609.153786:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757609.153786:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757609.153799:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659665152/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757609.153809:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757609.153814:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153820:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757609.153824:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787757609.153835:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787757609.153840:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757609.153844:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757609.153848:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757609.153851:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757609.153853:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757609.153857:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:2.0:1787757609.153859:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10663 00000100:00000001:1.0:1787757609.153864:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757609.153866:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757609.153868:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757609.153879:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757609.153883:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153883:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757609.153902:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757609.153904:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305337025 00000020:00000001:2.0:1787757609.153906:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757609.153911:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757609.153917:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757609.153918:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757609.153923:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757609.153926:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757609.153929:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305337025 00000100:00080000:2.0:1787757609.153936:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757609.153939:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757609.153941:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757609.153948:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757609.153953:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757609.153957:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757609.153957:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153971:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757609.153972:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.153982:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:2.0:1787757609.153983:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757609.153991:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.153996:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757609.153998:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757609/real 1787757609] req@ffff8ffb47a38000 x1874588659665024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757609.154005:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757609.154010:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757609.154016:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757609.154017:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000400:00000001:1.0:1787757609.154026:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:2.0:1787757609.154026:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757609.154036:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154038:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757609.154043:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757609.154044:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154047:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154053:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:2.0:1787757609.154055:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659665280/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757609.154057:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757609.154073:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659665024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757609.154094:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757609.154103:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154109:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757609.154112:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154115:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757609.154118:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757609.154123:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757609.154125:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757609.154129:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757609.154130:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757609.154133:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757609.154135:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757609.154138:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757609.154140:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757609.154143:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757609.154150:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10663 00000100:00080000:1.0:1787757609.154153:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757609.154160:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757609.154165:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757609.154175:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757609.154177:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757609.154179:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757609.154213:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659665024:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757609.154218:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154219:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757609.154221:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757609.154225:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757609.154234:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757609.154237:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757609.154240:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757609.154244:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757609.154245:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000020:00000001:2.0:1787757609.154252:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757609.154253:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000020:00000001:2.0:1787757609.154256:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757609.154257:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757609.154264:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757609.154265:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757609.154279:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757609.154284:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757609.154286:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757609.154290:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.154290:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757609.154311:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787757609.154313:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757609.154315:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757609.154321:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154323:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787757609.154333:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757609.154334:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757609.154343:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757609.154347:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757609.154349:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757609.154359:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.154360:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757609.154371:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757609.154373:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:2.0:1787757609.154383:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757609.154389:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665152:192.168.201.118@tcp:38:kworker.0 00000008:00000001:2.0:1787757609.154394:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.154402:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154403:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757609.154413:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757609.154415:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.154416:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757609.154419:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757609.154420:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757609.154425:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:2.0:1787757609.154426:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154429:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:2.0:1787757609.154434:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757609.154440:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787757609.154463:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757609.154472:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757609.154480:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757609.154489:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154490:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757609.154496:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757609.154499:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154502:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757609.154510:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757609.154512:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659665408/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757609.154517:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305336334 00000100:00000001:1.0:1787757609.154526:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305337025 00000100:00000001:1.0:1787757609.154538:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305336334 00000100:00000001:1.0:1787757609.154548:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305337025 00000100:00000001:1.0:1787757609.154559:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757609.154566:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757609.154568:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757609.154570:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787757609.154574:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757609.154581:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757609/real 1787757609] req@ffff8ffb47a38380 x1874588659665152/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757609.154583:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757609.154597:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757609.154600:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757609.154606:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757609.154609:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757609.154611:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154612:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757609.154614:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757609.154617:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659665152/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757609.154625:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154627:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757609.154629:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154630:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757609.154631:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154632:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757609.154643:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757609.154645:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757609.154648:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757609.154649:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757609.154652:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665152:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757609.154654:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757609.154656:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757609.154657:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.154658:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757609.154660:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757609.154662:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757609.154664:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757609.154666:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757609.154668:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757609.154669:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.154670:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757609.154673:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154677:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757609.154677:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757609.154680:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757609.154688:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757609.154691:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757609.154692:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757609.154693:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154694:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757609.154696:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154699:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305336331 00000100:00000001:1.0:1787757609.154700:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305337025 00000100:00000001:1.0:1787757609.154702:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305336331 00000100:00000001:1.0:1787757609.154704:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305337025 00000100:00000001:1.0:1787757609.154705:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757609.154706:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154708:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757609.154710:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757609/real 1787757609] req@ffff8ffb47a38a80 x1874588659665280/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757609.154714:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757609.154715:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757609.154717:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757609.154718:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757609.154719:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154720:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757609.154721:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757609.154723:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659665280/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757609.154727:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154728:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757609.154730:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154731:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757609.154731:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154732:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757609.154741:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757609.154742:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757609.154745:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757609.154746:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757609.154748:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757609.154751:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154752:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757609.154753:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757609.154754:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757609.154755:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757609.154756:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757609.154757:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757609.154759:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757609.154760:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757609.154761:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757609.154762:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154763:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757609.154764:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757609.154765:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757609.154766:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757609.154766:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757609.154767:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757609.154768:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757609.154769:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757609.154769:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757609.154770:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154771:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154774:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757609.154774:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154776:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757609.154777:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757609.154778:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757609.154779:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757609.154780:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.154781:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757609.154782:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757609.154784:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757609.154786:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757609.154788:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757609.154789:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757609.154790:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757609.154792:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757609.154793:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154796:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757609.154797:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757609.154799:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757609.154805:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757609.154808:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757609.154809:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757609.154810:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154811:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757609.154813:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154815:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305336331 00000100:00000001:1.0:1787757609.154817:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305337025 00000100:00000001:1.0:1787757609.154818:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305336331 00000100:00000001:1.0:1787757609.154820:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305337025 00000100:00000001:1.0:1787757609.154821:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757609.154822:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154823:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757609.154825:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757609/real 1787757609] req@ffff8ffb47a3a680 x1874588659665408/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757609.154830:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757609.154830:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757609.154832:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757609.154833:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757609.154834:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154835:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757609.154836:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757609.154838:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659665408/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757609.154842:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154844:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757609.154845:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154846:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757609.154847:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154848:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757609.154857:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757609.154859:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757609.154861:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757609.154862:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757609.154864:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757609.154866:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154867:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757609.154868:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757609.154869:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757609.154870:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757609.154870:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757609.154871:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757609.154872:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757609.154873:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757609.154875:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757609.154876:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154877:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757609.154879:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757609.154879:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757609.154881:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757609.154882:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757609.729287:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757609.729293:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757609.729295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757609.729299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757609.729302:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757609.729305:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757609.729307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757609.729309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757609.729311:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757609.921260:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757609.921267:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757609.921272:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757609.921273:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757609.921308:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757609.921311:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757609.921314:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757610.753270:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757610.753276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757610.753278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757610.753282:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757610.753285:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757610.753288:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757610.753290:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757610.753293:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757610.753295:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757610.945383:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757610.945392:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757610.945400:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757610.945403:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757610.945526:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757610.945534:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757610.945541:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757611.777355:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757611.777367:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757611.777371:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757611.777379:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757611.777384:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757611.777390:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757611.777393:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757611.777422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757611.777427:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757611.969317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757611.969326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757611.969334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757611.969340:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757611.969343:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757611.969348:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757611.969353:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757611.969356:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757611.969360:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757611.969363:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757611.969460:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757611.969468:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757611.969474:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757612.801351:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757612.801367:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757612.801372:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757612.801410:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757612.801416:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757612.801422:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757612.801426:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757612.801430:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757612.801435:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757612.993365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757612.993399:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757612.993412:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757612.993417:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757612.993547:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757612.993556:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757612.993562:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757613.825344:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757613.825354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757613.825409:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757613.825419:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757613.825426:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757613.825432:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757613.825436:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757613.825439:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757613.825443:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757614.017344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757614.017381:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757614.017393:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757614.017396:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757614.017483:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757614.017495:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757614.017505:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757614.273427:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757614.273440:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757614.273443:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757614.273447:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757614.273453:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757614.273463:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10669 00000100:00080000:3.0:1787757614.273478:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757614.273482:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757614.273484:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757614.273490:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757614.273493:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757614.273497:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757614.273500:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757614.273503:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757614.273504:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757614.273510:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757614.273514:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757614.273517:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757614.273521:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757614.273525:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273528:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757614.273531:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757614.273543:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757614.273545:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273548:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757614.273552:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757614.273557:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273571:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757614.273582:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659665536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757614.273596:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757614.273602:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273631:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757614.273636:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757614.273640:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757614.273642:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757614.273644:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757614.273656:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757614.273660:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10669 00000100:00080000:3.0:1787757614.273664:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757614.273667:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757614.273669:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757614.273672:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757614.273674:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757614.273676:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757614.273679:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757614.273681:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757614.273683:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757614.273687:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757614.273690:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757614.273693:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757614.273700:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757614.273703:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273706:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757614.273713:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757614.273718:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.273722:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757614.273722:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.273731:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757614.273733:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757614.273738:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757614.273739:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.273746:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757614.273747:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757614.273755:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757614.273756:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757614.273761:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273768:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757614.273775:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659665664/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757614.273780:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757614.273787:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757614.273794:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757614.273796:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273802:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273810:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757614.273811:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659665536:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757614.273817:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757614.273822:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757614.273828:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787757614.273832:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757614.273833:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757614.273841:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757614.273842:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273851:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757614.273852:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757614.273862:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10669 02000000:00000001:1.0:1787757614.273864:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757614.273866:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757614.273873:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757614.273875:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757614.273878:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757614.273881:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757614.273883:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757614.273886:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757614.273889:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757614.273891:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757614.273892:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757614.273898:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00080000:3.0:1787757614.273903:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757614.273910:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757614.273910:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757614.273916:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757614.273920:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757614.273924:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273926:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757614.273929:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757614.273932:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273935:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757614.273940:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757614.273942:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273945:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757614.273948:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757614.273951:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.273958:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757614.273969:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659665792/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757614.273973:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757614.273995:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757614.273998:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757614.274000:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.274008:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274012:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757614.274012:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757614.274021:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757614.274022:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757614.274027:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757614.274032:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757614.274033:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757614.274041:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757614.274045:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10669 00000100:00000001:1.0:1787757614.274049:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757614.274051:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757614.274058:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757614.274062:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757614.274066:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305342145 00000020:00000001:3.0:1787757614.274066:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757614.274070:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757614.274073:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757614.274079:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757614.274082:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757614.274084:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757614.274088:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787757614.274089:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757614.274098:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757614.274101:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757614.274104:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305342145 02000000:00000001:3.0:1787757614.274105:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757614.274114:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274116:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757614.274117:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757614.274125:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757614.274128:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787757614.274132:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274135:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.274138:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274142:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787757614.274148:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757614.274152:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.274157:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:1.0:1787757614.274158:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757614/real 1787757614] req@ffff8ffb45c31500 x1874588659665536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757614.274160:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757614.274165:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757614.274174:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757614.274251:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659665920/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757614.274252:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757614.274257:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757614.274267:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757614.274269:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757614.274274:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757614.274278:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757614.274283:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274285:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757614.274291:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757614.274292:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757614.274304:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659665536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757614.274332:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274337:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757614.274341:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274344:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757614.274345:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274348:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757614.274377:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757614.274384:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757614.274391:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757614.274395:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757614.274400:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659665536:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757614.274406:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274409:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757614.274411:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757614.274415:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757614.274420:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757614.274423:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757614.274426:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757614.274430:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757614.274433:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757614.274436:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757614.274438:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274441:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274449:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757614.274452:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274455:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757614.274457:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757614.274459:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757614.274461:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757614.274464:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757614.274467:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757614.274469:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757614.274472:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757614.274477:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665664:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757614.274481:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757614.274484:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757614.274487:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757614.274489:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757614.274493:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274502:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757614.274504:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757614.274510:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757614.274530:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757614.274536:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757614.274539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757614.274541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274544:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757614.274547:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274553:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305341454 00000100:00000001:1.0:1787757614.274556:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305342145 00000100:00000001:1.0:1787757614.274560:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305341454 00000100:00000001:1.0:1787757614.274563:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305342145 00000100:00000001:1.0:1787757614.274565:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757614.274568:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274570:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757614.274576:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757614/real 1787757614] req@ffff8ffb45c30e00 x1874588659665664/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757614.274586:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757614.274588:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757614.274591:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757614.274593:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757614.274595:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274597:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757614.274599:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757614.274604:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659665664/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757614.274622:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274624:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757614.274628:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274630:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757614.274631:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274633:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757614.274647:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757614.274650:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757614.274654:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757614.274656:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757614.274660:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665664:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757614.274665:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757614.274667:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757614.274669:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757614.274672:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757614.274674:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757614.274676:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757614.274680:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665792:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757614.274683:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757614.274685:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757614.274687:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757614.274689:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757614.274692:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274698:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757614.274700:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757614.274705:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757614.274717:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757614.274722:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757614.274724:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757614.274726:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274729:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757614.274732:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274735:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305341451 00000100:00000001:1.0:1787757614.274739:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305342145 00000100:00000001:1.0:1787757614.274741:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305341451 00000100:00000001:1.0:1787757614.274744:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305342145 00000100:00000001:1.0:1787757614.274747:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757614.274749:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274751:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757614.274756:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757614/real 1787757614] req@ffff8ffb45c30a80 x1874588659665792/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757614.274765:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757614.274766:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757614.274769:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757614.274771:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757614.274773:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274775:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757614.274777:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757614.274781:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659665792/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757614.274790:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274792:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757614.274795:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274797:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757614.274798:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274800:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757614.274813:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757614.274816:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757614.274820:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757614.274823:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757614.274827:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665792:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757614.274832:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757614.274834:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757614.274836:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757614.274838:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757614.274840:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757614.274842:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757614.274846:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757614.274849:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757614.274851:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757614.274852:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757614.274854:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757614.274858:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274862:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757614.274864:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757614.274868:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757614.274879:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757614.274884:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757614.274886:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757614.274888:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274890:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757614.274893:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274897:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305341451 00000100:00000001:1.0:1787757614.274900:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305342145 00000100:00000001:1.0:1787757614.274903:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305341451 00000100:00000001:1.0:1787757614.274906:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305342145 00000100:00000001:1.0:1787757614.274908:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757614.274911:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.274913:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757614.274917:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757614/real 1787757614] req@ffff8ffb45c33b80 x1874588659665920/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757614.274934:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757614.274936:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757614.274939:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757614.274941:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757614.274943:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274945:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757614.274946:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757614.274950:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659665920/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757614.274958:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274961:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757614.274964:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274967:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757614.274969:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.274972:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757614.274992:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757614.274997:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757614.275003:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757614.275007:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757614.275013:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659665920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757614.275020:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.275024:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757614.275026:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757614.275029:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757614.275034:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757614.275037:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757614.275040:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757614.275044:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757614.275047:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757614.275051:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757614.275053:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.275058:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757614.275060:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757614.275062:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757614.275065:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757614.275067:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757614.275070:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757614.275073:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757614.275075:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757614.275076:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757614.275078:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.275081:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757614.275082:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757614.275084:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757614.275087:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757614.275088:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757614.275089:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757614.275091:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757614.275092:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757614.275094:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757614.275096:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.275098:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757614.275103:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757614.275105:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757614.275107:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757614.275109:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757614.849397:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757614.849409:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757614.849412:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757614.849420:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757614.849426:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757614.849432:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757614.849436:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757614.849440:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757614.849444:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757615.041334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757615.041367:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757615.041373:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757615.041382:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757615.041392:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757615.041396:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757615.041401:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757615.041411:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757615.041419:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757615.041422:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757615.041483:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757615.041490:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757615.041494:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757615.873372:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757615.873384:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757615.873388:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757615.873395:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757615.873401:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757615.873408:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757615.873412:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757615.873416:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757615.873419:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757616.065358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757616.065368:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757616.065378:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757616.065380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757616.065482:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757616.065490:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757616.065495:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757616.897407:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757616.897417:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757616.897420:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757616.897427:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757616.897432:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757616.897437:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757616.897440:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757616.897443:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757616.897447:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757617.089395:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757617.089404:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757617.089412:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757617.089414:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757617.089466:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757617.089476:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757617.089483:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757617.921358:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757617.921368:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757617.921370:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757617.921377:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757617.921393:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757617.921398:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757617.921402:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757617.921405:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757617.921408:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757618.113322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757618.113330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757618.113337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757618.113340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757618.113487:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757618.113493:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757618.113498:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757618.945316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757618.945325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757618.945328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757618.945334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757618.945339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757618.945345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757618.945348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757618.945352:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757618.945355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757619.137311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757619.137320:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757619.137328:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757619.137331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757619.137449:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757619.137455:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757619.137460:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757619.393298:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757619.393311:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757619.393314:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757619.393319:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757619.393325:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757619.393334:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10674 00000100:00080000:3.0:1787757619.393348:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757619.393352:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757619.393355:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757619.393361:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757619.393364:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757619.393368:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757619.393371:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757619.393375:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757619.393377:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757619.393383:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757619.393387:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757619.393390:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757619.393394:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757619.393398:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393401:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757619.393404:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757619.393417:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757619.393419:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393424:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757619.393427:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757619.393432:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393447:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757619.393459:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659666048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757619.393475:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757619.393480:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393503:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757619.393507:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757619.393512:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757619.393514:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757619.393516:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757619.393520:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757619.393523:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10674 00000100:00080000:3.0:1787757619.393527:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757619.393530:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757619.393532:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757619.393535:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757619.393537:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757619.393540:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757619.393544:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757619.393547:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757619.393550:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757619.393555:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757619.393559:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757619.393561:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757619.393568:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757619.393572:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393574:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757619.393581:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787757619.393582:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757619.393590:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.393591:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757619.393594:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.393599:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757619.393605:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757619.393606:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757619.393612:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.393614:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393617:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757619.393623:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757619.393627:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757619.393628:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.393635:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757619.393639:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.393643:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757619.393652:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659666176/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757619.393659:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659666048:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757619.393676:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757619.393682:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757619.393687:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757619.393688:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.393696:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757619.393699:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.393704:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393711:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757619.393721:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757619.393729:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757619.393730:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757619.393736:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:3.0:1787757619.393739:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757619.393744:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757619.393748:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757619.393758:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757619.393766:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10674 00000100:00080000:3.0:1787757619.393777:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757619.393781:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757619.393785:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757619.393789:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757619.393791:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757619.393794:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000800:00000100:1.0:1787757619.393801:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787757619.393803:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757619.393812:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757619.393815:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757619.393821:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757619.393824:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757619.393832:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.393833:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757619.393835:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757619.393840:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757619.393844:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757619.393846:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757619.393854:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.393858:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393859:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757619.393870:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305347265 00000008:00100000:3.0:1787757619.393870:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757619.393884:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393889:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757619.393898:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757619.393901:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.393904:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757619.393907:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757619.393909:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757619.393919:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.393922:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305347265 00000100:00000001:3.0:1787757619.393933:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.393934:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757619.393948:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757619.393955:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757619.393957:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659666304/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757619.393960:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757619.393974:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757619/real 1787757619] req@ffff8ffb45c31c00 x1874588659666048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757619.393988:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757619.393997:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.394006:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757619.394010:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:3.0:1787757619.394012:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000400:00000001:1.0:1787757619.394018:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:3.0:1787757619.394022:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757619.394026:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757619.394029:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757619.394036:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787757619.394036:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757619.394044:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757619.394046:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757619.394051:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757619.394058:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10674 00000100:00000001:1.0:1787757619.394059:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757619.394073:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757619.394076:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659666048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757619.394082:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757619.394089:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757619.394099:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757619.394102:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757619.394106:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787757619.394110:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757619.394112:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394118:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:3.0:1787757619.394120:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757619.394122:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394129:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000020:00000001:3.0:1787757619.394134:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757619.394135:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757619.394148:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757619.394149:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757619.394157:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757619.394158:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757619.394166:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00080000:3.0:1787757619.394176:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1787757619.394178:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659666048:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757619.394225:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394229:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757619.394234:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757619.394236:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.394239:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757619.394244:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757619.394249:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:3.0:1787757619.394251:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787757619.394256:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:3.0:1787757619.394258:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757619.394287:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757619.394291:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757619.394295:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000008:00100000:3.0:1787757619.394298:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757619.394302:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000008:00000001:3.0:1787757619.394306:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394309:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757619.394314:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394315:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394322:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757619.394326:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757619.394331:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787757619.394332:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394338:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.394340:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757619.394347:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757619.394348:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757619.394356:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757619.394357:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394362:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757619.394367:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757619.394370:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.394371:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.394374:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757619.394380:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00100000:3.0:1787757619.394387:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659666432/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757619.394388:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757619.394403:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666176:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757619.394417:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757619.394423:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757619.394424:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757619.394430:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757619.394432:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.394434:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757619.394442:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.394444:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394458:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757619.394461:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757619.394466:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757619.394488:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757619.394494:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757619.394498:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757619.394500:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394503:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757619.394506:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394511:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305346574 00000100:00000001:1.0:1787757619.394515:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305347265 00000100:00000001:1.0:1787757619.394518:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305346574 00000100:00000001:1.0:1787757619.394521:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305347265 00000100:00000001:1.0:1787757619.394524:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757619.394526:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394528:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757619.394534:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757619/real 1787757619] req@ffff8ffb45c33b80 x1874588659666176/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757619.394545:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757619.394546:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757619.394549:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757619.394551:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757619.394552:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394555:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757619.394556:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757619.394561:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659666176/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757619.394570:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394572:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757619.394575:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394577:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757619.394579:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394582:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757619.394596:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757619.394600:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757619.394604:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757619.394607:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757619.394612:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666176:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757619.394616:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757619.394618:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757619.394620:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.394622:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757619.394624:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757619.394627:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757619.394630:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666304:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757619.394635:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757619.394637:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757619.394639:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.394641:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757619.394644:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394650:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757619.394651:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757619.394656:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757619.394668:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757619.394672:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757619.394675:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757619.394677:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394679:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757619.394681:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394685:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305346571 00000100:00000001:1.0:1787757619.394688:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305347265 00000100:00000001:1.0:1787757619.394691:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305346571 00000100:00000001:1.0:1787757619.394694:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305347265 00000100:00000001:1.0:1787757619.394699:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757619.394701:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394704:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757619.394709:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757619/real 1787757619] req@ffff8ffb45c30a80 x1874588659666304/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757619.394718:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757619.394719:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757619.394722:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757619.394723:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757619.394725:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394727:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757619.394729:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757619.394733:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659666304/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757619.394742:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394745:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757619.394747:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394749:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757619.394751:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394753:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757619.394765:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757619.394768:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757619.394771:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757619.394773:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757619.394777:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666304:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757619.394781:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394783:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757619.394785:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757619.394787:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757619.394790:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757619.394792:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757619.394794:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757619.394797:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757619.394799:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757619.394801:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757619.394803:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394805:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757619.394807:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757619.394808:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757619.394812:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757619.394813:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757619.394815:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757619.394817:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757619.394818:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757619.394820:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757619.394822:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394824:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394827:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757619.394829:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394831:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757619.394833:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757619.394834:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757619.394836:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757619.394838:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.394840:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757619.394842:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757619.394845:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757619.394849:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757619.394853:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757619.394854:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757619.394857:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757619.394858:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757619.394861:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394866:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757619.394868:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757619.394872:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757619.394884:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757619.394888:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757619.394890:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757619.394892:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394895:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757619.394898:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394902:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305346571 00000100:00000001:1.0:1787757619.394905:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305347265 00000100:00000001:1.0:1787757619.394908:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305346571 00000100:00000001:1.0:1787757619.394911:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305347265 00000100:00000001:1.0:1787757619.394913:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757619.394915:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.394917:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757619.394922:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757619/real 1787757619] req@ffff8ffb45c30e00 x1874588659666432/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757619.394931:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757619.394932:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757619.394935:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757619.394937:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757619.394938:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394940:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757619.394942:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757619.394947:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659666432/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757619.394955:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394957:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757619.394960:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394962:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757619.394963:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394965:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757619.394977:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757619.394980:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757619.394983:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757619.394986:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757619.394990:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757619.394995:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.394997:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757619.394999:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757619.395001:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757619.395002:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757619.395004:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757619.395005:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757619.395008:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757619.395009:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757619.395011:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757619.395013:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.395015:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757619.395018:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757619.395020:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757619.395023:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757619.395025:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757619.969365:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757619.969378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757619.969381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757619.969389:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757619.969394:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757619.969399:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757619.969402:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757619.969406:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757619.969409:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757620.161361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757620.161371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757620.161384:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757620.161387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757620.161459:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757620.161465:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757620.161472:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757620.993319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757620.993328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757620.993331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757620.993338:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757620.993342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757620.993347:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757620.993351:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757620.993354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757620.993358:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757621.185325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757621.185333:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757621.185342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757621.185347:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757621.185351:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757621.185356:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757621.185360:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757621.185364:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757621.185367:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757621.185370:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757621.185437:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757621.185444:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757621.185449:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757622.017300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757622.017310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757622.017313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757622.017320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757622.017324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757622.017329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757622.017332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757622.017335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757622.017339:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757622.209349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757622.209359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757622.209369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757622.209372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757622.209466:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757622.209471:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757622.209476:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757623.041349:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757623.041360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757623.041363:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757623.041370:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757623.041375:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757623.041379:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757623.041382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757623.041385:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757623.041389:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757623.233370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757623.233379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757623.233386:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757623.233388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757623.233485:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757623.233491:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757623.233513:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757624.065414:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757624.065427:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757624.065431:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757624.065440:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757624.065445:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757624.065451:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757624.065472:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757624.065477:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757624.065482:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757624.257413:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757624.257433:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757624.257438:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757624.257445:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757624.257456:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757624.257461:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757624.257465:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757624.257471:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757624.257480:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757624.257484:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757624.257580:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757624.257589:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757624.257595:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757624.513371:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757624.513386:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787757624.513390:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757624.513397:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757624.513404:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757624.513415:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10679 00000100:00080000:0.0:1787757624.513432:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757624.513437:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757624.513440:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757624.513446:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757624.513451:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757624.513455:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787757624.513459:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787757624.513462:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757624.513464:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757624.513470:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757624.513474:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757624.513477:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757624.513482:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757624.513487:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513490:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757624.513493:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757624.513513:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757624.513517:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513522:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757624.513526:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757624.513534:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513554:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757624.513568:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ed80 x1874588659666560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757624.513584:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757624.513590:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513619:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757624.513625:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757624.513629:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787757624.513632:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757624.513634:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757624.513638:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757624.513641:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10679 00000100:00080000:0.0:1787757624.513645:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757624.513648:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757624.513650:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757624.513653:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757624.513654:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757624.513657:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787757624.513661:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757624.513664:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757624.513666:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757624.513672:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757624.513676:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757624.513678:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757624.513687:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757624.513691:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513693:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757624.513697:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:0.0:1787757624.513701:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757624.513710:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.513710:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757624.513723:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.513725:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757624.513735:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:0.0:1787757624.513737:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757624.513740:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757624.513742:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513760:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757624.513765:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757624.513765:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757624.513779:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757624.513779:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757624.513799:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513801:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757624.513816:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790fb80 x1874588659666688/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757624.513823:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659666560:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757624.513843:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757624.513850:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:0.0:1787757624.513859:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757624.513859:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513867:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757624.513873:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:0.0:1787757624.513876:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757624.513882:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757624.513888:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787757624.513889:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513892:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757624.513896:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757624.513903:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757624.513910:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10679 00000100:00080000:0.0:1787757624.513914:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757624.513917:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757624.513919:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757624.513923:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:0.0:1787757624.513924:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757624.513933:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000020:00000001:0.0:1787757624.513935:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757624.513949:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757624.513952:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:0.0:1787757624.513958:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757624.513963:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757624.513965:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757624.513971:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757624.513977:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757624.513981:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757624.513987:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757624.513991:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.513994:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757624.513998:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757624.514002:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.514004:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757624.514010:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757624.514013:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.514020:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757624.514025:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000800:00000100:1.0:1787757624.514028:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:0.0:1787757624.514029:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.514042:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757624.514052:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790d180 x1874588659666816/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757624.514062:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757624.514065:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.514068:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757624.514074:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757624.514076:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:0.0:1787757624.514089:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757624.514093:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757624.514096:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757624.514097:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757624.514106:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757624.514106:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514119:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:0.0:1787757624.514127:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10679 00000100:00000001:1.0:1787757624.514132:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757624.514141:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757624.514143:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305352385 00000100:00000001:1.0:1787757624.514161:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757624.514273:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305352386 00000100:00000001:0.0:1787757624.514280:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757624.514286:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757624.514291:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:0.0:1787757624.514293:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757624.514298:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757624.514309:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:0.0:1787757624.514313:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757624.514316:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757624.514319:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757624.514324:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:0.0:1787757624.514327:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757624.514331:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.514335:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514342:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:0.0:1787757624.514343:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757624.514360:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757624.514373:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757624.514375:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757624/real 1787757624] req@ffff8ffb4790ed80 x1874588659666560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757624.514385:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757624.514394:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757624.514412:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.514429:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757624.514441:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:0.0:1787757624.514442:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:1.0:1787757624.514448:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:0.0:1787757624.514449:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.514459:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757624.514462:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757624.514465:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:0.0:1787757624.514467:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514477:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757624.514480:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514486:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757624.514494:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790e680 x1874588659666944/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757624.514499:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757624.514509:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757624.514519:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757624.514524:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757624.514529:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757624.514538:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659666560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757624.514559:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514566:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757624.514571:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514574:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757624.514576:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514580:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757624.514583:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757624.514588:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757624.514594:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757624.514600:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757624.514605:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659666560:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757624.514612:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514615:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757624.514618:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757624.514621:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757624.514627:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757624.514630:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757624.514633:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757624.514639:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757624.514641:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757624.514645:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757624.514647:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514650:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514659:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757624.514661:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514664:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757624.514667:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757624.514669:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757624.514672:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757624.514674:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757624.514678:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757624.514681:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757624.514684:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757624.514689:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666688:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757624.514693:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757624.514697:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757624.514699:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757624.514701:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757624.514706:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514715:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757624.514717:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:1.0:1787757624.514723:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757624.514746:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757624.514753:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757624.514757:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757624.514776:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514784:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757624.514789:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514794:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305351694 00000100:00000001:1.0:1787757624.514799:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305352385 00000100:00000001:1.0:1787757624.514803:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305351694 00000100:00000001:1.0:1787757624.514808:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305352386 00000100:00000001:1.0:1787757624.514812:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757624.514816:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514820:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757624.514828:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757624/real 1787757624] req@ffff8ffb4790fb80 x1874588659666688/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757624.514847:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757624.514850:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757624.514855:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757624.514859:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757624.514862:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514866:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757624.514869:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757624.514877:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659666688/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757624.514892:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514897:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757624.514902:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514905:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757624.514908:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.514911:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757624.514928:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757624.514931:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757624.514936:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757624.514939:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757624.514945:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666688:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757624.514950:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757624.514953:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757624.514955:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757624.514958:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757624.514961:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757624.514963:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757624.514967:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666816:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757624.514971:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757624.514974:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757624.514977:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757624.514979:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757624.514984:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.514993:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757624.514995:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:1.0:1787757624.515002:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757624.515028:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757624.515037:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757624.515042:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757624.515045:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.515048:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757624.515053:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.515060:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305351691 00000100:00000001:1.0:1787757624.515066:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305352385 00000100:00000001:1.0:1787757624.515070:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305351691 00000100:00000001:1.0:1787757624.515074:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305352386 00000100:00000001:1.0:1787757624.515078:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757624.515082:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.515086:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757624.515095:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757624/real 1787757624] req@ffff8ffb4790d180 x1874588659666816/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757624.515118:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757624.515119:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757624.515123:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757624.515126:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757624.515128:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515130:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757624.515132:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757624.515137:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659666816/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757624.515146:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515149:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757624.515152:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515154:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757624.515156:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515158:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757624.515233:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757624.515240:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757624.515247:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757624.515251:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757624.515259:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666816:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757624.515268:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757624.515271:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757624.515274:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757624.515279:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757624.515283:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757624.515287:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757624.515293:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757624.515300:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757624.515304:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757624.515309:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757624.515311:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757624.515316:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.515325:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757624.515326:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:1.0:1787757624.515331:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757624.515350:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757624.515357:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757624.515360:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757624.515362:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.515364:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757624.515368:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.515372:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305351692 00000100:00000001:1.0:1787757624.515375:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305352385 00000100:00000001:1.0:1787757624.515379:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305351692 00000100:00000001:1.0:1787757624.515381:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305352386 00000100:00000001:1.0:1787757624.515384:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757624.515387:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.515389:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757624.515394:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757624/real 1787757624] req@ffff8ffb4790e680 x1874588659666944/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757624.515403:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757624.515405:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757624.515407:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757624.515410:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757624.515411:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515414:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757624.515416:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757624.515421:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659666944/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757624.515429:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515432:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757624.515435:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515437:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757624.515439:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515442:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757624.515456:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757624.515459:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757624.515464:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757624.515469:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757624.515476:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659666944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757624.515483:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515487:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757624.515490:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757624.515492:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757624.515497:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757624.515500:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757624.515503:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757624.515507:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757624.515510:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757624.515514:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757624.515516:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515520:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757624.515522:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757624.515525:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757624.515529:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757624.515531:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757624.515534:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757624.515537:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757624.515539:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757624.515543:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757624.515546:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515549:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757624.515552:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757624.515554:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757624.515557:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757624.515559:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757624.515562:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757624.515566:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757624.515569:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757624.515573:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757624.515574:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515577:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757624.515583:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757624.515585:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757624.515588:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757624.515590:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757625.089396:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757625.089408:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757625.089412:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757625.089420:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757625.089425:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757625.089432:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757625.089436:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757625.089440:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757625.089444:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757625.281393:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757625.281404:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757625.281412:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757625.281414:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757625.281490:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757625.281499:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757625.281505:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757626.113392:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757626.113405:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757626.113409:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757626.113417:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757626.113423:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757626.113429:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757626.113432:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757626.113436:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757626.113441:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757626.305316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757626.305325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757626.305335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757626.305338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757626.305470:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757626.305478:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757626.305485:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757627.137318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757627.137327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757627.137330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757627.137337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757627.137341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757627.137346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757627.137350:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757627.137353:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757627.137357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757627.329307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757627.329319:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757627.329332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757627.329336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757627.329467:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757627.329477:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757627.329484:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757628.161274:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757628.161283:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757628.161286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757628.161292:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757628.161297:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757628.161301:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757628.161304:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757628.161307:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757628.161311:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757628.353314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757628.353322:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757628.353330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757628.353332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757628.353429:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757628.353435:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757628.353440:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757629.185282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757629.185291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757629.185294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757629.185300:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757629.185305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757629.185309:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.185312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757629.185315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.185319:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757629.377307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757629.377314:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757629.377323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757629.377325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757629.377459:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757629.377465:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757629.377470:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757629.633363:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757629.633379:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757629.633384:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757629.633391:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757629.633400:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757629.633413:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10684 00000100:00080000:3.0:1787757629.633429:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757629.633433:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757629.633437:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757629.633443:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757629.633447:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757629.633451:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757629.633455:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757629.633458:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757629.633462:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757629.633468:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757629.633474:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757629.633479:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757629.633484:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757629.633490:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.633493:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757629.633497:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757629.633509:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757629.633512:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.633517:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757629.633521:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757629.633526:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.633542:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757629.633555:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659667072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757629.633570:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757629.633577:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.633614:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757629.633621:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757629.633626:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757629.633630:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757629.633634:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757629.633640:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757629.633646:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10684 00000100:00080000:3.0:1787757629.633653:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757629.633657:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757629.633660:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757629.633663:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757629.633666:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757629.633670:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757629.633674:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757629.633680:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757629.633680:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757629.633683:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757629.633693:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757629.633695:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757629.633698:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757629.633701:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757629.633709:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757629.633711:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.633714:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757629.633723:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:3.0:1787757629.633726:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757629.633732:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:3.0:1787757629.633733:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.633740:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.633741:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.633752:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757629.633758:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757629.633761:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757629.633765:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757629.633766:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.633775:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757629.633778:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757629.633783:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757629.633787:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659667072:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757629.633794:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.633800:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757629.633810:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757629.633816:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.633821:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757629.633830:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.633853:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757629.633857:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787757629.633865:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757629.633920:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1787757629.633928:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659667200/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757629.633938:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757629.633948:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757629.633949:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757629.633955:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.633956:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.633962:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757629.633965:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.633972:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757629.633973:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757629.633983:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757629.633987:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305357505 00000100:00000001:3.0:1787757629.633987:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757629.633993:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757629.634001:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757629.634005:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10684 00000100:00000001:0.0:1787757629.634011:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787757629.634013:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757629.634024:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757629.634028:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757629.634030:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305357505 00000020:00000001:3.0:1787757629.634038:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757629.634045:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757629.634046:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:3.0:1787757629.634050:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757629.634057:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757629.634065:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634071:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757629.634109:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757629/real 1787757629] req@ffff8ffb45c31500 x1874588659667072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757629.634130:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757629.634133:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757629.634140:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757629.634144:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757629.634147:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.634152:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757629.634156:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757629.634165:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659667072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757629.634225:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.634231:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757629.634236:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.634240:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757629.634242:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.634245:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757629.634249:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757629.634253:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757629.634260:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757629.634265:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757629.634271:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659667072:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757629.634279:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.634282:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757629.634287:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757629.634291:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757629.634297:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757629.634300:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757629.634303:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757629.634309:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757629.634311:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757629.634314:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757629.634316:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.634319:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.634328:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757629.634330:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634334:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757629.634336:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757629.634338:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757629.634340:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757629.634342:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.634345:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757629.634348:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757629.634352:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757629.634359:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667200:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757629.634367:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757629.634371:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757629.634374:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.634378:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757629.634384:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757629.634397:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757629.634398:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757629.634406:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:3.0:1787757629.634409:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757629.634416:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757629.634419:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757629.634432:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757629.634438:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757629.634442:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757629.634448:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757629.634453:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.634457:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000800:00000100:0.0:1787757629.634463:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1787757629.634465:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757629.634473:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.634477:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634481:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757629.634486:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757629.634492:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757629.634499:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634503:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757629.634510:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634519:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305356814 00000100:00000001:0.0:1787757629.634526:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305357505 00000100:00000001:0.0:1787757629.634531:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305356814 00000100:00000001:0.0:1787757629.634536:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305357505 00000100:00000001:0.0:1787757629.634542:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757629.634547:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634553:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757629.634565:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757629/real 1787757629] req@ffff8ffb45c30700 x1874588659667200/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757629.634587:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757629.634591:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757629.634596:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634603:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757629.634604:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757629.634612:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757629.634615:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787757629.634617:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634624:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757629.634630:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634633:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757629.634640:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757629.634647:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659667328/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757629.634656:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659667200/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757629.634680:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757629.634689:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757629.634689:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634700:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757629.634703:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634706:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757629.634710:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757629.634712:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757629.634716:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757629.634724:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757629.634725:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757629.634732:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757629.634736:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757629.634741:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757629.634748:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10684 00000100:00000001:0.0:1787757629.634751:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757629.634757:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757629.634759:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757629.634766:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757629.634772:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757629.634773:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000020:00000001:3.0:1787757629.634781:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757629.634784:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:3.0:1787757629.634787:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:0.0:1787757629.634794:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667200:192.168.201.118@tcp:38:kworker.0 00000020:00000001:3.0:1787757629.634796:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757629.634805:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757629.634808:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757629.634811:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757629.634815:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757629.634822:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787757629.634822:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757629.634832:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757629.634834:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757629.634838:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757629.634844:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757629.634848:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757629.634851:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00080000:3.0:1787757629.634851:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757629.634856:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757629.634861:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757629.634864:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634866:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757629.634876:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757629.634878:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757629.634884:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757629.634886:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757629.634887:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757629.634892:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757629.634900:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757629.634901:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634903:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757629.634910:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757629.634911:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757629.634919:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757629.634920:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757629.634925:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787757629.634927:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.634932:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.634934:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757629.634943:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667328:192.168.201.118@tcp:8:kworker.0 02000000:00000001:3.0:1787757629.634945:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757629.634951:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.634953:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757629.634956:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757629.634959:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757629.634961:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757629.634965:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757629.634968:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.634974:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757629.634979:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.634986:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757629.634994:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659667456/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757629.635004:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757629.635008:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757629.635012:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757629.635014:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757629.635023:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635028:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757629.635065:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757629.635077:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757629.635112:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757629.635116:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635120:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757629.635127:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635134:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305356811 00000100:00000001:0.0:1787757629.635140:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305357505 00000100:00000001:0.0:1787757629.635144:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305356811 00000100:00000001:0.0:1787757629.635148:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305357505 00000100:00000001:0.0:1787757629.635153:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757629.635157:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635161:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757629.635171:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757629/real 1787757629] req@ffff8ffb45c32a00 x1874588659667328/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757629.635224:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757629.635227:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757629.635236:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757629.635241:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757629.635245:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635248:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757629.635250:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757629.635257:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659667328/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757629.635268:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635272:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757629.635277:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635280:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757629.635282:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635285:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757629.635303:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757629.635306:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757629.635310:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757629.635313:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757629.635317:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667328:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757629.635322:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635325:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757629.635327:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757629.635329:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757629.635333:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757629.635335:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757629.635338:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757629.635341:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757629.635342:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757629.635345:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757629.635347:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635350:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635355:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757629.635356:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635359:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757629.635361:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757629.635363:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757629.635365:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757629.635367:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.635371:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757629.635373:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757629.635376:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757629.635380:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757629.635384:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757629.635386:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757629.635388:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757629.635391:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757629.635395:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635402:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757629.635405:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757629.635409:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757629.635428:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757629.635434:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757629.635437:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757629.635439:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635441:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757629.635444:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635449:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305356812 00000100:00000001:0.0:1787757629.635452:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305357505 00000100:00000001:0.0:1787757629.635455:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305356812 00000100:00000001:0.0:1787757629.635458:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305357505 00000100:00000001:0.0:1787757629.635461:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757629.635464:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635466:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757629.635471:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757629/real 1787757629] req@ffff8ffb45c30000 x1874588659667456/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757629.635482:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757629.635483:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757629.635486:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757629.635488:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757629.635490:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635492:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757629.635494:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757629.635498:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659667456/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757629.635507:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635510:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757629.635512:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635514:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757629.635516:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635518:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757629.635532:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757629.635537:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757629.635543:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757629.635547:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757629.635553:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757629.635559:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635563:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757629.635565:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757629.635568:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757629.635572:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757629.635575:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757629.635578:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757629.635581:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757629.635585:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757629.635589:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757629.635591:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635595:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757629.635602:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757629.635605:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757629.635609:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757629.635612:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757630.209270:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757630.209279:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757630.209281:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757630.209287:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757630.209293:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757630.209298:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757630.209301:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757630.209304:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757630.209308:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757630.401331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757630.401338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757630.401346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757630.401351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757630.401406:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757630.401420:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757630.401430:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757631.233334:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757631.233343:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757631.233346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757631.233352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757631.233357:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757631.233362:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757631.233366:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757631.233369:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757631.233372:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757631.425306:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757631.425315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757631.425322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757631.425328:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757631.425331:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757631.425335:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757631.425340:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757631.425343:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757631.425346:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757631.425350:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757631.425499:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757631.425508:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757631.425515:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757632.257296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757632.257305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757632.257309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757632.257315:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757632.257320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757632.257325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757632.257328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757632.257331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757632.257335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757632.449294:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757632.449301:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757632.449309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757632.449311:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757632.449410:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757632.449416:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757632.449421:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757633.281289:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757633.281298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757633.281301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757633.281307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757633.281312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757633.281317:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757633.281320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757633.281323:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757633.281326:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757633.473380:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757633.473387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757633.473394:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757633.473396:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757633.473488:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757633.473506:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757633.473511:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757634.305363:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757634.305375:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757634.305379:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757634.305389:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757634.305396:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757634.305403:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.305407:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757634.305412:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.305418:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757634.497362:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757634.497372:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757634.497376:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757634.497380:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757634.497387:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757634.497390:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757634.497394:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757634.497397:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757634.497402:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757634.497404:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757634.497533:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757634.497546:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757634.497556:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757634.753324:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757634.753343:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757634.753347:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757634.753354:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757634.753361:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757634.753373:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10689 00000100:00080000:3.0:1787757634.753389:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757634.753393:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757634.753396:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757634.753401:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757634.753404:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757634.753408:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757634.753412:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757634.753415:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757634.753417:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757634.753423:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757634.753427:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757634.753430:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757634.753435:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757634.753438:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753441:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757634.753445:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757634.753457:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757634.753460:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753464:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757634.753468:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757634.753473:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753488:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757634.753517:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659667584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757634.753532:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757634.753538:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753557:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757634.753561:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757634.753566:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757634.753569:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757634.753571:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757634.753575:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757634.753578:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10689 00000100:00080000:3.0:1787757634.753583:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757634.753587:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757634.753589:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757634.753593:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757634.753595:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757634.753599:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757634.753602:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757634.753605:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757634.753608:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757634.753613:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757634.753615:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757634.753619:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757634.753620:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757634.753626:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757634.753630:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757634.753633:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.753636:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757634.753640:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757634.753643:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757634.753648:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757634.753651:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757634.753658:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:3.0:1787757634.753659:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753667:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.753669:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757634.753672:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757634.753678:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.753682:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753683:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757634.753688:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757634.753695:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753702:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757634.753704:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659667584:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787757634.753715:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659667712/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757634.753724:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757634.753735:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757634.753736:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757634.753746:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753746:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.753760:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757634.753761:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757634.753770:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757634.753775:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757634.753776:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757634.753785:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757634.753789:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757634.753793:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757634.753797:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10689 00000100:00080000:3.0:1787757634.753802:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757634.753804:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757634.753806:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757634.753807:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757634.753813:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000020:00000001:3.0:1787757634.753815:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757634.753822:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757634.753824:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787757634.753827:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757634.753833:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757634.753836:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757634.753838:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757634.753843:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757634.753847:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757634.753850:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757634.753854:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757634.753857:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753860:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757634.753864:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757634.753867:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753869:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757634.753877:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000800:00000100:0.0:1787757634.753879:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757634.753884:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753892:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757634.753896:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757634.753898:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757634.753901:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.753907:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757634.753913:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.753916:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.753918:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757634.753927:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.753936:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305362625 00000100:00100000:3.0:1787757634.753938:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659667840/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757634.753952:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757634.753960:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757634.753964:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305362625 00000100:00000001:3.0:1787757634.753971:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.753975:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757634.753985:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757634.753985:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757634.754024:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757634.754031:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757634.754035:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757634.754044:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757634.754044:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757634.754051:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757634.754056:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:0.0:1787757634.754065:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757634/real 1787757634] req@ffff8ffb45c33800 x1874588659667584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757634.754072:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10689 00000100:00080000:3.0:1787757634.754092:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000400:00000001:0.0:1787757634.754096:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757634.754101:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:0.0:1787757634.754102:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757634.754111:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757634.754115:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757634.754122:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000020:00000001:3.0:1787757634.754124:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757634.754127:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757634.754133:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757634.754134:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757634.754140:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757634.754143:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:0.0:1787757634.754151:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659667584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757634.754155:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757634.754169:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757634.754176:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757634.754178:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757634.754223:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757634.754229:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754232:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757634.754234:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754238:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757634.754244:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757634.754253:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1787757634.754254:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757634.754268:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757634.754269:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757634.754276:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1787757634.754281:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:0.0:1787757634.754285:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659667584:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757634.754292:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757634.754300:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.754304:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757634.754312:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757634.754315:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754326:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00000001:3.0:1787757634.754326:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754331:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757634.754338:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.754339:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757634.754348:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757634.754353:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:3.0:1787757634.754356:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000020:00000001:0.0:1787757634.754357:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757634.754366:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.754368:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757634.754378:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757634.754378:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757634.754386:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757634.754388:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757634.754394:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757634.754394:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754404:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757634.754412:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754415:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757634.754421:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754428:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:3.0:1787757634.754429:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659667968/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757634.754432:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757634.754437:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757634.754442:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757634.754447:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.754456:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757634.754461:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757634.754462:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757634.754471:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.754473:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757634.754486:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757634.754487:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667712:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757634.754500:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757634.754504:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757634.754506:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.754509:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757634.754515:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754525:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757634.754528:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787757634.754533:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757634.754553:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757634.754560:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757634.754564:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757634.754566:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754568:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757634.754572:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754577:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305361934 00000100:00000001:0.0:1787757634.754581:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305362625 00000100:00000001:0.0:1787757634.754584:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305361934 00000100:00000001:0.0:1787757634.754587:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305362625 00000100:00000001:0.0:1787757634.754590:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757634.754592:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754595:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757634.754602:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757634/real 1787757634] req@ffff8ffb45c31180 x1874588659667712/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757634.754613:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757634.754615:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757634.754618:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757634.754620:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757634.754622:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754624:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757634.754626:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757634.754631:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659667712/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757634.754639:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754642:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757634.754645:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754647:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757634.754648:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754650:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757634.754671:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757634.754677:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757634.754683:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757634.754688:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757634.754694:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667712:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757634.754701:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757634.754704:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757634.754707:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.754712:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757634.754715:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757634.754720:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757634.754726:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667840:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757634.754739:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757634.754743:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757634.754746:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.754749:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757634.754755:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754768:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757634.754771:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:0.0:1787757634.754776:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757634.754795:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757634.754802:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757634.754805:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757634.754807:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754809:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757634.754813:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754817:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305361931 00000100:00000001:0.0:1787757634.754820:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305362625 00000100:00000001:0.0:1787757634.754823:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305361931 00000100:00000001:0.0:1787757634.754826:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305362625 00000100:00000001:0.0:1787757634.754829:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757634.754831:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754834:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757634.754839:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757634/real 1787757634] req@ffff8ffb45c33480 x1874588659667840/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757634.754848:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757634.754850:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757634.754852:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757634.754854:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757634.754856:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754858:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757634.754860:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757634.754865:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659667840/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757634.754882:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754885:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757634.754888:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754890:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757634.754891:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754893:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757634.754906:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757634.754909:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757634.754913:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757634.754915:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757634.754919:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667840:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757634.754924:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754926:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757634.754928:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757634.754930:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757634.754933:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757634.754935:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757634.754936:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757634.754939:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757634.754941:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757634.754943:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757634.754945:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754947:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757634.754949:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757634.754950:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757634.754953:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757634.754954:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757634.754956:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757634.754958:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757634.754959:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757634.754960:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757634.754962:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754964:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.754968:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757634.754971:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.754974:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757634.754975:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757634.754977:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757634.754979:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757634.754981:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.754983:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757634.754985:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757634.754988:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757634.755029:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667968:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757634.755033:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757634.755036:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757634.755038:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757634.755040:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757634.755044:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.755052:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757634.755054:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757634.755059:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757634.755076:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757634.755082:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757634.755085:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757634.755087:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.755089:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757634.755093:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.755097:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305361931 00000100:00000001:0.0:1787757634.755100:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305362625 00000100:00000001:0.0:1787757634.755103:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305361931 00000100:00000001:0.0:1787757634.755106:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305362625 00000100:00000001:0.0:1787757634.755108:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757634.755111:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.755113:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757634.755118:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757634/real 1787757634] req@ffff8ffb45c30000 x1874588659667968/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757634.755135:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757634.755137:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757634.755140:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757634.755142:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757634.755144:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.755146:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757634.755147:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757634.755152:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659667968/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757634.755160:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.755163:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757634.755166:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.755168:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757634.755170:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.755172:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757634.755213:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757634.755217:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757634.755221:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757634.755224:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757634.755228:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659667968:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757634.755233:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.755235:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757634.755237:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757634.755239:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757634.755242:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757634.755244:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757634.755246:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757634.755249:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757634.755250:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757634.755253:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757634.755255:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.755258:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757634.755262:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757634.755264:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757634.755266:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757634.755269:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757635.329296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757635.329307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757635.329310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757635.329316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757635.329321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757635.329326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757635.329330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757635.329333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757635.329336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757635.521332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757635.521340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757635.521350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757635.521352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757635.521436:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757635.521442:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757635.521447:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757636.353333:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757636.353346:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757636.353350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757636.353359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757636.353366:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757636.353372:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757636.353377:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757636.353380:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757636.353384:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757636.545315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757636.545324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757636.545333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757636.545335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757636.545411:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757636.545417:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757636.545422:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757637.377304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757637.377316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757637.377320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757637.377329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757637.377335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757637.377342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757637.377347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757637.377354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757637.377359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757637.569329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757637.569338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757637.569347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757637.569350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757637.569487:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757637.569493:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757637.569498:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757638.401331:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757638.401342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757638.401345:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757638.401353:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757638.401358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757638.401364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757638.401368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757638.401371:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757638.401376:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757638.593391:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757638.593402:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757638.593410:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757638.593412:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757638.593557:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757638.593568:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757638.593576:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757639.425329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757639.425342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757639.425345:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757639.425354:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757639.425360:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757639.425368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.425373:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757639.425379:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.425385:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757639.617383:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757639.617394:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757639.617404:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757639.617406:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757639.617515:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757639.617524:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757639.617530:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757639.873298:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757639.873311:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757639.873315:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757639.873320:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757639.873340:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757639.873353:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10694 00000100:00080000:3.0:1787757639.873370:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757639.873377:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757639.873381:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757639.873388:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757639.873393:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757639.873398:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757639.873405:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757639.873409:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757639.873412:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757639.873421:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757639.873426:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757639.873430:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757639.873435:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757639.873439:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873442:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757639.873446:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757639.873460:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757639.873462:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873466:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757639.873470:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757639.873475:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873490:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757639.873502:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659668096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757639.873517:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757639.873523:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873545:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757639.873549:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757639.873554:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757639.873557:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757639.873559:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757639.873564:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757639.873568:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10694 00000100:00080000:3.0:1787757639.873574:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757639.873577:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757639.873580:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757639.873583:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757639.873585:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757639.873588:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757639.873590:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757639.873593:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757639.873594:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757639.873598:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757639.873601:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757639.873605:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757639.873612:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757639.873615:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873618:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757639.873625:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757639.873630:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873633:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757639.873639:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757639.873641:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873643:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757639.873645:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757639.873648:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873654:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757639.873660:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659668224/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757639.873668:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757639.873672:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757639.873681:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.873683:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757639.873690:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.873691:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757639.873699:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787757639.873700:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757639.873708:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757639.873720:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757639.873724:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757639.873727:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757639.873730:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787757639.873749:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757639.873749:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757639.873767:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757639.873777:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10694 00000100:00100000:0.0:1787757639.873779:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659668096:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757639.873788:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757639.873794:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757639.873799:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757639.873804:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757639.873808:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757639.873811:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757639.873824:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:0.0:1787757639.873825:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787757639.873835:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757639.873841:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757639.873843:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757639.873858:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757639.873866:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757639.873869:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757639.873881:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757639.873882:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757639.873887:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757639.873895:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.873898:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757639.873925:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757639.873934:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757639.873941:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873946:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757639.873952:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757639.873958:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873962:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757639.873973:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757639.873977:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.873981:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757639.873984:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757639.873989:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757639.874000:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757639.874004:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874021:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:3.0:1787757639.874027:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659668352/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757639.874032:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757639.874041:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874053:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757639.874066:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757639.874067:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.874074:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874082:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305367745 00000100:00000001:3.0:1787757639.874085:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757639.874095:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757639.874100:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757639.874103:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757639.874104:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757639.874120:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757639.874126:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305367745 00000100:00000001:3.0:1787757639.874128:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757639.874146:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10694 00000100:00000001:0.0:1787757639.874148:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787757639.874159:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757639.874160:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757639.874169:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757639.874171:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.874175:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757639.874181:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757639.874247:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757639.874252:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:3.0:1787757639.874252:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757639.874265:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:0.0:1787757639.874270:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757639/real 1787757639] req@ffff8ffb45c32a00 x1874588659668096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757639.874274:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757639.874285:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:0.0:1787757639.874314:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:3.0:1787757639.874315:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787757639.874320:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757639.874331:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874333:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757639.874339:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787757639.874343:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757639.874344:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874356:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757639.874360:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757639.874362:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757639.874372:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.874376:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787757639.874380:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659668096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757639.874384:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757639.874406:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.874424:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757639.874444:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757639.874447:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757639.874449:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874456:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757639.874456:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757639.874468:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757639.874470:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757639.874476:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874479:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757639.874489:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874497:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757639.874498:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874502:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757639.874514:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757639.874517:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659668480/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757639.874531:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757639.874541:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757639.874546:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757639.874547:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757639.874559:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757639.874570:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659668096:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757639.874590:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874596:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757639.874601:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757639.874608:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757639.874618:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757639.874623:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757639.874628:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757639.874634:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757639.874637:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757639.874642:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757639.874644:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874647:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874658:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757639.874661:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874665:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757639.874667:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757639.874670:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757639.874673:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757639.874675:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757639.874679:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757639.874682:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757639.874685:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757639.874691:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668224:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757639.874696:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757639.874699:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757639.874702:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757639.874704:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757639.874709:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874718:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757639.874720:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757639.874728:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757639.874748:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757639.874755:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757639.874758:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757639.874760:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874762:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757639.874766:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874772:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305367054 00000100:00000001:0.0:1787757639.874776:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305367745 00000100:00000001:0.0:1787757639.874780:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305367054 00000100:00000001:0.0:1787757639.874783:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305367745 00000100:00000001:0.0:1787757639.874786:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757639.874788:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874790:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757639.874796:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757639/real 1787757639] req@ffff8ffb45c30700 x1874588659668224/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757639.874807:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757639.874808:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757639.874812:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757639.874814:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757639.874816:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874818:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757639.874820:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757639.874825:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659668224/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757639.874843:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874846:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757639.874849:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874851:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757639.874853:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.874855:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757639.874877:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757639.874880:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757639.874884:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757639.874887:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757639.874891:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668224:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757639.874896:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757639.874927:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757639.874931:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757639.874935:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757639.874938:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757639.874941:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757639.874946:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668352:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757639.874950:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757639.874954:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757639.874956:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757639.874958:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757639.874963:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.874972:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757639.874974:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787757639.874980:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757639.874995:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757639.875002:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757639.875005:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757639.875007:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.875010:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757639.875014:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.875018:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305367051 00000100:00000001:0.0:1787757639.875022:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305367745 00000100:00000001:0.0:1787757639.875025:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305367051 00000100:00000001:0.0:1787757639.875028:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305367745 00000100:00000001:0.0:1787757639.875030:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757639.875033:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.875035:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757639.875040:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757639/real 1787757639] req@ffff8ffb45c31500 x1874588659668352/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757639.875049:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757639.875051:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757639.875053:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757639.875056:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757639.875057:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875060:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757639.875062:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757639.875066:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659668352/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757639.875075:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875078:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757639.875081:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875083:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757639.875084:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875087:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757639.875107:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757639.875110:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757639.875113:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757639.875116:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757639.875122:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668352:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757639.875127:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757639.875129:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757639.875131:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757639.875134:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757639.875136:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757639.875138:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757639.875142:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757639.875145:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757639.875147:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757639.875150:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757639.875152:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757639.875156:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.875161:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757639.875163:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:0.0:1787757639.875168:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757639.875179:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757639.875184:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757639.875220:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757639.875222:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.875225:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757639.875230:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.875234:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305367052 00000100:00000001:0.0:1787757639.875238:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305367745 00000100:00000001:0.0:1787757639.875241:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305367052 00000100:00000001:0.0:1787757639.875244:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305367745 00000100:00000001:0.0:1787757639.875247:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757639.875249:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.875251:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757639.875256:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757639/real 1787757639] req@ffff8ffb45c30e00 x1874588659668480/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757639.875275:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757639.875277:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757639.875280:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757639.875282:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757639.875284:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875286:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757639.875288:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757639.875293:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659668480/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757639.875302:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875304:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757639.875307:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875309:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757639.875310:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875312:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757639.875331:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757639.875334:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757639.875337:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757639.875340:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757639.875343:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757639.875348:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875350:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757639.875352:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757639.875354:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757639.875357:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757639.875359:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757639.875361:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757639.875364:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757639.875366:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757639.875369:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757639.875371:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875374:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757639.875375:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757639.875377:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757639.875380:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757639.875382:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757639.875384:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757639.875386:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757639.875387:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757639.875389:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757639.875391:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875392:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757639.875394:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757639.875395:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757639.875396:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757639.875398:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757639.875399:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757639.875401:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757639.875402:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757639.875404:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757639.875405:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875407:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757639.875412:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757639.875413:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757639.875416:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757639.875418:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757640.449435:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757640.449447:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757640.449451:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757640.449459:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757640.449465:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757640.449471:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757640.449475:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757640.449479:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757640.449483:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757640.641438:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757640.641450:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757640.641460:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757640.641462:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757640.641538:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757640.641545:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757640.641551:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757641.473412:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757641.473423:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757641.473427:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757641.473437:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757641.473443:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757641.473464:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757641.473468:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757641.473473:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757641.473477:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757641.665343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757641.665352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757641.665359:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757641.665363:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757641.665367:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757641.665371:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757641.665375:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757641.665378:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757641.665381:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757641.665384:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757641.665459:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757641.665465:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757641.665470:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757642.497366:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757642.497382:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757642.497387:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757642.497397:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757642.497403:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757642.497409:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757642.497413:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757642.497417:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757642.497422:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757642.689368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757642.689379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757642.689389:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757642.689391:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757642.689489:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757642.689497:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757642.689504:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757643.521335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757643.521345:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757643.521348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757643.521354:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757643.521359:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757643.521365:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757643.521369:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757643.521373:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757643.521376:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757643.713347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757643.713359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757643.713369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757643.713372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757643.713478:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757643.713487:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757643.713494:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757644.545305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757644.545314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757644.545317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757644.545324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757644.545330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757644.545337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757644.545342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757644.545347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757644.545352:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757644.737300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757644.737311:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757644.737315:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757644.737319:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757644.737327:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757644.737330:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757644.737333:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757644.737336:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757644.737341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757644.737344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757644.737495:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757644.737501:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757644.737505:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757644.993275:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757644.993286:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787757644.993289:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757644.993293:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757644.993297:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757644.993305:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10699 00000100:00080000:0.0:1787757644.993322:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757644.993325:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757644.993327:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757644.993330:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757644.993332:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757644.993334:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787757644.993337:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787757644.993338:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757644.993339:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757644.993343:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757644.993344:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757644.993346:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757644.993349:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757644.993351:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993353:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757644.993355:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757644.993365:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757644.993367:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993369:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757644.993372:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757644.993376:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993386:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757644.993393:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ca80 x1874588659668608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757644.993401:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757644.993404:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993419:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757644.993421:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757644.993424:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787757644.993425:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757644.993426:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757644.993428:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757644.993430:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10699 00000100:00080000:0.0:1787757644.993432:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757644.993433:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757644.993434:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757644.993435:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757644.993437:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757644.993438:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787757644.993440:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757644.993441:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757644.993442:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757644.993444:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757644.993446:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757644.993447:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757644.993451:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757644.993453:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993454:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757644.993460:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757644.993462:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993463:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757644.993466:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757644.993467:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993468:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757644.993469:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:0.0:1787757644.993470:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757644.993472:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993477:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993477:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757644.993483:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:0.0:1787757644.993484:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f480 x1874588659668736/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757644.993488:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757644.993493:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757644.993493:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993497:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993501:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.993501:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:0.0:1787757644.993506:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757644.993508:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:0.0:1787757644.993509:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757644.993511:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757644.993512:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757644.993514:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757644.993517:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10699 00000100:00080000:0.0:1787757644.993518:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757644.993520:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757644.993521:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757644.993523:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757644.993525:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757644.993525:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757644.993528:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787757644.993531:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757644.993533:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757644.993534:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757644.993537:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659668608:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1787757644.993538:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757644.993542:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757644.993545:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757644.993549:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757644.993550:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757644.993551:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993554:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757644.993556:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757644.993557:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00000001:0.0:1787757644.993558:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993562:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.993563:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757644.993564:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757644.993567:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.993569:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:0.0:1787757644.993570:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757644.993572:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757644.993574:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.993578:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993579:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757644.993586:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790dc00 x1874588659668864/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757644.993591:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757644.993593:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993594:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757644.993596:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757644.993598:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757644.993599:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757644.993600:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:0.0:1787757644.993601:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757644.993605:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:0.0:1787757644.993606:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757644.993609:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10699 00000100:00080000:0.0:1787757644.993613:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757644.993614:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787757644.993616:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757644.993618:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757644.993619:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757644.993620:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757644.993622:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:0.0:1787757644.993623:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757644.993624:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757644.993625:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757644.993627:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757644.993629:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757644.993630:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757644.993632:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757644.993634:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993635:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757644.993636:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757644.993637:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993638:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757644.993641:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757644.993642:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993643:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757644.993644:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757644.993645:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757644.993649:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757644.993651:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790c000 x1874588659668992/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757644.993656:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757644.993660:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757644.993661:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757644.993662:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993677:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757644.993683:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757644.993686:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993689:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757644.993695:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993699:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305372865 00000100:00000001:1.0:1787757644.993710:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757644.993716:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305372865 00000100:00000001:1.0:1787757644.993722:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757644.993726:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757644.993730:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993733:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757644.993740:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757644/real 1787757644] req@ffff8ffb4790ca80 x1874588659668608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757644.993752:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757644.993754:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757644.993759:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757644.993761:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757644.993763:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.993766:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757644.993768:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757644.993773:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659668608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757644.993782:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.993786:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757644.993789:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.993791:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757644.993792:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.993795:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757644.993797:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757644.993801:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757644.993806:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757644.993823:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757644.993828:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659668608:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757644.993834:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.993836:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757644.993838:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757644.993841:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757644.993846:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757644.993849:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757644.993851:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757644.993856:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757644.993858:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757644.993860:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757644.993862:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.993865:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.993872:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757644.993875:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993877:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757644.993879:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757644.993881:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757644.993882:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757644.993884:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.993886:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757644.993889:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757644.993891:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757644.993895:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668736:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757644.993899:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757644.993901:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757644.993903:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.993905:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757644.993908:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993915:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757644.993917:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:1.0:1787757644.993922:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757644.993936:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757644.993941:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757644.993944:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757644.993946:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993947:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757644.993950:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993955:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305372173 00000100:00000001:1.0:1787757644.993958:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305372865 00000100:00000001:1.0:1787757644.993961:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305372173 00000100:00000001:1.0:1787757644.993963:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305372865 00000100:00000001:1.0:1787757644.993966:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757644.993968:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.993971:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757644.993976:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757644/real 1787757644] req@ffff8ffb4790f480 x1874588659668736/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757644.993986:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757644.993988:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757644.993990:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757644.993992:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757644.993994:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.993996:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757644.993997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757644.994001:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659668736/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757644.994009:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994012:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757644.994015:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994016:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757644.994018:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994020:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757644.994035:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757644.994038:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757644.994042:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757644.994044:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757644.994048:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668736:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757644.994053:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757644.994054:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757644.994056:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.994058:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757644.994061:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757644.994063:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757644.994067:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668864:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757644.994070:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757644.994072:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757644.994073:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.994075:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757644.994078:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994084:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757644.994085:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133080064 : -123165576471552 : ffff8ffb4790dc00) 00000100:00000001:1.0:1787757644.994090:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757644.994102:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757644.994106:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757644.994108:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757644.994110:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994112:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757644.994114:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994118:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305372170 00000100:00000001:1.0:1787757644.994121:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305372865 00000100:00000001:1.0:1787757644.994123:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305372170 00000100:00000001:1.0:1787757644.994125:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305372865 00000100:00000001:1.0:1787757644.994127:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757644.994130:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994132:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757644.994136:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757644/real 1787757644] req@ffff8ffb4790dc00 x1874588659668864/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757644.994146:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757644.994147:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757644.994150:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757644.994152:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757644.994153:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994156:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757644.994157:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757644.994162:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790dc00 x1874588659668864/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757644.994177:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994179:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757644.994212:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994214:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757644.994216:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994218:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757644.994232:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757644.994236:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757644.994240:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757644.994243:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757644.994247:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668864:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757644.994251:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757644.994253:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757644.994254:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.994256:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757644.994259:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757644.994261:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757644.994264:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757644.994267:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757644.994270:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757644.994271:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757644.994273:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757644.994277:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994282:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757644.994284:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133072896 : -123165576478720 : ffff8ffb4790c000) 00000100:00000001:1.0:1787757644.994288:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757644.994302:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757644.994307:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757644.994309:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757644.994311:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994313:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757644.994315:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994319:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305372171 00000100:00000001:1.0:1787757644.994323:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305372865 00000100:00000001:1.0:1787757644.994325:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305372171 00000100:00000001:1.0:1787757644.994328:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305372865 00000100:00000001:1.0:1787757644.994330:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757644.994333:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994335:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757644.994340:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757644/real 1787757644] req@ffff8ffb4790c000 x1874588659668992/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757644.994349:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757644.994351:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757644.994354:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757644.994356:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757644.994357:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994359:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757644.994360:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757644.994365:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c000 x1874588659668992/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757644.994373:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994376:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757644.994378:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994380:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757644.994382:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994384:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757644.994397:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757644.994400:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757644.994405:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757644.994407:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757644.994412:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659668992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757644.994416:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994418:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757644.994420:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757644.994422:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757644.994426:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757644.994428:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757644.994430:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757644.994434:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757644.994436:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757644.994438:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757644.994440:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994442:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757644.994443:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757644.994445:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757644.994448:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757644.994449:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757644.994450:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757644.994453:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757644.994455:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757644.994457:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757644.994459:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994462:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757644.994464:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757644.994465:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757644.994467:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757644.994469:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757644.994470:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757644.994472:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757644.994473:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757644.994475:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757644.994477:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994479:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757644.994483:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757644.994485:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757644.994487:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757644.994489:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757645.569268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757645.569275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757645.569276:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757645.569286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757645.569288:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757645.569292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757645.569293:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757645.569296:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757645.569298:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757645.761274:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757645.761282:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757645.761289:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757645.761291:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757645.761326:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757645.761330:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757645.761333:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757646.593305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757646.593314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757646.593317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757646.593324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757646.593329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757646.593334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757646.593338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757646.593342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757646.593345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757646.785314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757646.785323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757646.785346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757646.785349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757646.785470:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757646.785481:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757646.785489:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757647.617319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757647.617329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757647.617332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757647.617338:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757647.617343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757647.617348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757647.617351:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757647.617355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757647.617359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757647.809348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757647.809356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757647.809363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757647.809365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757647.809441:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757647.809448:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757647.809453:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757648.641362:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757648.641374:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757648.641378:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757648.641386:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757648.641393:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757648.641400:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757648.641405:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757648.641410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757648.641414:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757648.833353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757648.833362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757648.833370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757648.833373:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757648.833511:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757648.833517:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757648.833523:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757649.665300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757649.665309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757649.665312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757649.665319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757649.665323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757649.665328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757649.665331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757649.665334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757649.665338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757649.857405:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757649.857414:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757649.857421:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757649.857423:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757649.857529:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757649.857535:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757649.857540:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757650.113381:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757650.113393:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757650.113396:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757650.113400:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757650.113407:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757650.113416:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10704 00000100:00080000:3.0:1787757650.113430:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757650.113435:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757650.113438:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757650.113443:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757650.113447:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757650.113451:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757650.113454:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757650.113457:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757650.113459:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757650.113465:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757650.113468:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.113472:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757650.113476:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757650.113480:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113482:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757650.113488:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.113504:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757650.113508:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113513:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757650.113518:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757650.113525:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113543:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757650.113561:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659669120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757650.113583:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757650.113590:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113612:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757650.113617:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757650.113622:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757650.113624:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757650.113627:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757650.113631:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757650.113636:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10704 00000100:00080000:3.0:1787757650.113644:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757650.113648:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757650.113650:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757650.113655:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757650.113657:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757650.113661:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757650.113664:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757650.113668:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757650.113668:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757650.113675:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757650.113679:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787757650.113685:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757650.113686:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757650.113693:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757650.113695:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.113698:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.113701:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757650.113708:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757650.113713:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757650.113715:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757650.113741:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.113746:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757650.113755:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659669120:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757650.113755:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757650.113767:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757650.113768:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:3.0:1787757650.113777:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757650.113778:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757650.113785:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113786:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757650.113792:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757650.113798:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757650.113803:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.113804:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113815:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757650.113817:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757650.113821:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.113831:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757650.113833:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.113838:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757650.113846:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:3.0:1787757650.113848:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659669248/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757650.113863:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757650.113868:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113873:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757650.113877:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757650.113882:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757650.113885:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757650.113888:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757650.113891:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757650.113896:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10704 00000800:00000100:1.0:1787757650.113906:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757650.113909:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757650.113915:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757650.113918:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757650.113922:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757650.113923:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:3.0:1787757650.113925:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757650.113932:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757650.113934:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:3.0:1787757650.113937:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757650.113941:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113943:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757650.113948:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757650.113950:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757650.113960:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757650.113962:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757650.113971:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305377985 00000100:00000001:3.0:1787757650.113972:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.113979:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757650.113984:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757650.113987:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757650.113988:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.113997:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757650.114002:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757650.114003:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305377985 00000008:00000001:3.0:1787757650.114011:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114015:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757650.114018:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114023:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757650.114030:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757650.114031:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.114038:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114040:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757650.114048:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757650.114053:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757650.114057:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757650/real 1787757650] req@ffff8ffb45c30a80 x1874588659669120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757650.114058:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.114070:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757650.114087:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659669376/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757650.114088:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757650.114094:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757650.114101:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757650.114109:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757650.114109:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757650.114117:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757650.114117:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114127:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757650.114128:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114131:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757650.114137:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757650.114144:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659669120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757650.114157:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757650.114163:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.114168:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757650.114172:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.114178:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757650.114180:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757650.114223:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.114228:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757650.114231:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757650.114243:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757650.114251:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1787757650.114254:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757650.114256:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757650.114269:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659669120:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757650.114271:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757650.114285:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757650.114288:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10704 00000100:00000001:1.0:1787757650.114291:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757650.114295:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757650.114301:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00080000:3.0:1787757650.114306:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757650.114307:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757650.114313:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757650.114318:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757650.114318:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757650.114328:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757650.114329:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757650.114333:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757650.114338:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000020:00000001:3.0:1787757650.114341:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757650.114342:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.114350:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757650.114352:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757650.114361:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000020:00000001:3.0:1787757650.114361:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757650.114366:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114374:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:3.0:1787757650.114376:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757650.114378:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757650.114388:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757650.114394:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757650.114395:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757650.114400:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757650.114407:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757650.114413:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757650.114416:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757650.114418:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757650.114426:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.114434:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757650.114436:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669248:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757650.114450:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757650.114450:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757650.114455:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757650.114460:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.114462:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757650.114463:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757650.114470:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.114475:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114480:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757650.114484:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757650.114486:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000008:00100000:3.0:1787757650.114495:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757650.114496:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00000001:3.0:1787757650.114508:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.114514:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757650.114521:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757650.114533:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757650.114534:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757650.114543:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114546:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757650.114551:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.114555:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757650.114559:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757650.114564:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757650.114570:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757650.114573:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114583:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305377294 00000100:00000001:3.0:1787757650.114591:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114594:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305377985 00000100:00000001:1.0:1787757650.114605:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305377294 00000100:00100000:3.0:1787757650.114612:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659669504/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757650.114614:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305377985 00000100:00000001:1.0:1787757650.114627:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757650.114639:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.114650:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757650.114651:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757650.114658:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757650.114667:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757650.114672:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757650/real 1787757650] req@ffff8ffb45c33b80 x1874588659669248/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757650.114696:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757650.114700:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757650.114708:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757650.114712:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757650.114715:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.114748:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757650.114751:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757650.114763:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659669248/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757650.114780:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.114785:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757650.114790:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.114794:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757650.114797:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.114801:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757650.114825:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757650.114832:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757650.114839:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757650.114843:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757650.114850:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669248:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757650.114859:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757650.114862:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757650.114865:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757650.114870:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757650.114873:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757650.114877:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757650.114884:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669376:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757650.114890:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757650.114894:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757650.114898:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757650.114901:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757650.114907:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114919:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757650.114923:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:1.0:1787757650.114930:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757650.114958:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757650.114969:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757650.114974:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757650.114978:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114982:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757650.114987:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.114994:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305377291 00000100:00000001:1.0:1787757650.115000:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305377985 00000100:00000001:1.0:1787757650.115005:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305377291 00000100:00000001:1.0:1787757650.115010:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305377985 00000100:00000001:1.0:1787757650.115014:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757650.115018:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.115022:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757650.115031:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757650/real 1787757650] req@ffff8ffb45c31c00 x1874588659669376/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757650.115049:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757650.115052:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757650.115057:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757650.115061:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757650.115064:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115068:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757650.115071:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757650.115079:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659669376/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757650.115093:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115097:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757650.115102:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115105:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757650.115107:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115111:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757650.115130:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757650.115134:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757650.115138:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757650.115141:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757650.115146:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669376:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757650.115152:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115154:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757650.115156:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757650.115159:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757650.115162:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757650.115165:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757650.115167:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757650.115170:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757650.115172:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757650.115174:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757650.115176:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115178:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757650.115180:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757650.115211:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757650.115215:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757650.115217:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757650.115218:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757650.115221:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757650.115222:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757650.115224:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757650.115226:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115229:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115234:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757650.115236:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.115239:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757650.115241:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757650.115243:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757650.115245:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757650.115247:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757650.115251:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757650.115253:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757650.115256:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757650.115260:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757650.115264:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757650.115267:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757650.115269:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757650.115272:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757650.115277:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.115284:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757650.115286:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757650.115291:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757650.115309:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757650.115315:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757650.115318:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757650.115320:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.115322:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757650.115326:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.115330:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305377292 00000100:00000001:1.0:1787757650.115334:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305377985 00000100:00000001:1.0:1787757650.115337:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305377292 00000100:00000001:1.0:1787757650.115340:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305377985 00000100:00000001:1.0:1787757650.115342:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757650.115345:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.115347:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757650.115352:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757650/real 1787757650] req@ffff8ffb45c30a80 x1874588659669504/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757650.115361:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757650.115363:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757650.115366:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757650.115368:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757650.115370:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115372:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757650.115374:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757650.115379:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659669504/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757650.115387:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115390:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757650.115393:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115395:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757650.115398:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115400:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757650.115414:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757650.115417:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757650.115421:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757650.115424:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757650.115428:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757650.115432:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115434:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757650.115435:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757650.115437:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757650.115439:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757650.115441:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757650.115442:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757650.115444:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757650.115446:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757650.115448:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757650.115449:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115451:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757650.115455:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757650.115457:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757650.115459:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757650.115461:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757650.689317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757650.689326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757650.689329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757650.689336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757650.689341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757650.689345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757650.689349:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757650.689352:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757650.689355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757650.881333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757650.881342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757650.881362:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757650.881367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757650.881417:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757650.881426:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757650.881436:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757651.713257:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757651.713263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757651.713265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757651.713269:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757651.713272:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757651.713276:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757651.713278:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757651.713280:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757651.713282:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757651.905255:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757651.905260:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757651.905265:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757651.905267:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757651.905269:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757651.905272:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757651.905274:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757651.905276:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757651.905278:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757651.905279:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757651.905319:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757651.905323:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757651.905325:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757652.737294:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757652.737303:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757652.737306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757652.737312:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757652.737317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757652.737322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757652.737325:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757652.737328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757652.737332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757652.929322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757652.929331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757652.929354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757652.929357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757652.929437:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757652.929443:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757652.929449:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757653.761341:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757653.761350:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757653.761353:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757653.761359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757653.761364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757653.761369:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757653.761372:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757653.761375:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757653.761378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757653.953372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757653.953380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757653.953387:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757653.953389:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757653.953469:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757653.953484:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757653.953489:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757654.785301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757654.785321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757654.785324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757654.785330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757654.785335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757654.785339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757654.785343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757654.785346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757654.785349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757654.977313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757654.977325:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757654.977331:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757654.977337:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757654.977345:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757654.977350:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757654.977355:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757654.977361:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757654.977370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757654.977374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757654.977471:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757654.977477:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757654.977482:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757655.233274:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757655.233285:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757655.233288:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757655.233292:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757655.233308:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757655.233316:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10709 00000100:00080000:3.0:1787757655.233328:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757655.233332:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757655.233334:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757655.233338:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757655.233342:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757655.233345:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757655.233348:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757655.233350:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757655.233352:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757655.233356:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757655.233359:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757655.233361:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757655.233365:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757655.233368:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233370:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757655.233372:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757655.233383:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757655.233385:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233388:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757655.233409:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757655.233415:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233429:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757655.233442:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659669632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757655.233456:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757655.233461:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233481:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757655.233486:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757655.233491:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757655.233493:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757655.233496:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757655.233500:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757655.233503:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10709 00000100:00080000:3.0:1787757655.233507:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757655.233509:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757655.233511:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757655.233515:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757655.233516:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757655.233519:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757655.233522:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757655.233525:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757655.233527:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757655.233530:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757655.233533:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757655.233535:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757655.233550:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757655.233554:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233557:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757655.233563:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757655.233568:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233571:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757655.233576:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757655.233578:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233581:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757655.233583:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757655.233585:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.233590:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757655.233592:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.233599:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:3.0:1787757655.233606:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659669760/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757655.233607:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757655.233613:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757655.233619:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757655.233657:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757655.233661:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757655.233680:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757655.233686:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757655.233687:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757655.233701:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659669632:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757655.233701:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.233711:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757655.233712:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757655.233717:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787757655.233724:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787757655.233725:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233751:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:1.0:1787757655.233752:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757655.233761:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757655.233764:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233768:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757655.233777:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10709 00000100:00080000:3.0:1787757655.233785:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757655.233788:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757655.233789:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757655.233795:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:3.0:1787757655.233796:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757655.233806:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787757655.233807:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757655.233812:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757655.233815:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757655.233818:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757655.233820:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757655.233822:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757655.233827:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757655.233830:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757655.233832:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757655.233838:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757655.233841:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233844:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757655.233847:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757655.233851:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233853:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757655.233860:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757655.233862:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757655.233867:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233870:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757655.233873:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757655.233877:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.233878:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757655.233910:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757655.233910:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.233913:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.233918:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:3.0:1787757655.233922:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659669888/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757655.233926:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233934:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757655.233937:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.233939:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757655.233941:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757655.233947:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757655.233949:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305383105 00000100:00000001:3.0:1787757655.233950:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757655.233956:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757655.233960:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757655.233963:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10709 00000100:00000001:1.0:1787757655.233965:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787757655.233968:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757655.233990:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757655.233993:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305383105 00000100:00000001:3.0:1787757655.233993:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757655.234004:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757655.234005:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:3.0:1787757655.234007:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757655.234015:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757655.234016:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787757655.234021:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757655.234024:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757655.234027:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757655.234031:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757655.234032:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757655.234042:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1787757655.234049:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757655/real 1787757655] req@ffff8ffb45c30e00 x1874588659669632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757655.234050:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757655.234059:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757655.234066:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757655.234076:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757655.234082:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757655.234082:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000400:00000001:1.0:1787757655.234090:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000008:00100000:3.0:1787757655.234091:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757655.234097:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234099:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757655.234102:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234105:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787757655.234111:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757655.234112:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757655.234118:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234120:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757655.234124:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757655.234126:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757655.234129:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757655.234134:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757655.234137:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659669632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757655.234142:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234160:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757655.234160:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659670016/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757655.234185:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757655.234193:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757655.234195:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757655.234196:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757655.234201:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234202:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234210:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757655.234210:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234242:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757655.234247:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757655.234252:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757655.234255:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757655.234258:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659669632:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757655.234264:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234266:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757655.234268:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757655.234270:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757655.234275:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757655.234277:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757655.234279:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757655.234283:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757655.234285:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757655.234287:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757655.234289:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234292:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234299:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757655.234300:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234303:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757655.234304:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757655.234306:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757655.234307:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757655.234309:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757655.234311:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757655.234313:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757655.234315:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757655.234318:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669760:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757655.234321:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757655.234323:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757655.234325:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757655.234327:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757655.234330:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234337:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757655.234339:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757655.234343:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757655.234356:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757655.234361:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757655.234363:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757655.234365:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234366:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757655.234369:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234373:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305382413 00000100:00000001:1.0:1787757655.234376:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305383105 00000100:00000001:1.0:1787757655.234378:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305382413 00000100:00000001:1.0:1787757655.234380:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305383105 00000100:00000001:1.0:1787757655.234383:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757655.234384:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234386:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757655.234390:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757655/real 1787757655] req@ffff8ffb45c31500 x1874588659669760/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757655.234398:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757655.234399:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757655.234401:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757655.234403:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757655.234404:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234406:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757655.234407:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757655.234411:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659669760/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757655.234417:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234419:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757655.234421:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234423:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757655.234424:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234426:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757655.234439:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757655.234442:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757655.234445:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757655.234447:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757655.234450:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669760:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757655.234453:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757655.234455:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757655.234456:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757655.234458:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757655.234459:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757655.234461:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757655.234463:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669888:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757655.234466:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757655.234467:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757655.234469:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757655.234470:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757655.234473:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234477:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757655.234479:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757655.234482:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757655.234491:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757655.234494:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757655.234496:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757655.234498:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234499:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757655.234501:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234504:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305382410 00000100:00000001:1.0:1787757655.234507:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305383105 00000100:00000001:1.0:1787757655.234509:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305382410 00000100:00000001:1.0:1787757655.234511:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305383105 00000100:00000001:1.0:1787757655.234515:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757655.234517:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234518:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757655.234522:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757655/real 1787757655] req@ffff8ffb45c30700 x1874588659669888/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757655.234529:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757655.234530:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757655.234532:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757655.234533:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757655.234534:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234536:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757655.234537:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757655.234540:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659669888/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757655.234547:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234549:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757655.234551:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234553:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757655.234554:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234555:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757655.234565:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757655.234567:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757655.234570:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757655.234572:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757655.234574:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659669888:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757655.234577:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757655.234578:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757655.234580:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757655.234582:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757655.234583:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757655.234585:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757655.234588:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757655.234591:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757655.234593:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757655.234594:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757655.234596:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757655.234598:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234602:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757655.234603:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787757655.234607:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757655.234615:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757655.234618:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757655.234620:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757655.234621:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234623:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757655.234625:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234647:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305382410 00000100:00000001:1.0:1787757655.234652:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305383105 00000100:00000001:1.0:1787757655.234655:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305382410 00000100:00000001:1.0:1787757655.234659:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305383105 00000100:00000001:1.0:1787757655.234662:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757655.234665:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234667:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757655.234673:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757655/real 1787757655] req@ffff8ffb45c32a00 x1874588659670016/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757655.234686:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757655.234688:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757655.234692:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757655.234695:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757655.234696:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234698:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757655.234700:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757655.234704:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659670016/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757655.234717:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234719:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757655.234722:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234723:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757655.234724:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234726:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757655.234737:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757655.234739:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757655.234741:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757655.234743:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757655.234746:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757655.234750:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234751:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757655.234753:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757655.234754:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757655.234757:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757655.234758:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757655.234759:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757655.234762:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757655.234763:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757655.234765:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757655.234766:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234768:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757655.234769:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757655.234770:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757655.234772:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757655.234773:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757655.234775:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757655.234776:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757655.234777:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757655.234778:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757655.234779:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234781:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757655.234782:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757655.234784:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757655.234785:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757655.234786:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757655.234787:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757655.234789:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757655.234790:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757655.234791:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757655.234793:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234794:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757655.234797:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757655.234799:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757655.234800:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757655.234802:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757655.809297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757655.809310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757655.809331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757655.809340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757655.809346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757655.809353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757655.809357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757655.809362:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757655.809367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757656.001304:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757656.001312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757656.001321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757656.001323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757656.001379:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757656.001383:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757656.001387:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757656.833280:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757656.833287:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757656.833288:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757656.833292:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757656.833295:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757656.833298:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757656.833300:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757656.833302:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757656.833312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757657.025326:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757657.025334:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757657.025342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757657.025344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757657.025471:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757657.025478:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757657.025482:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757657.857315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757657.857325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757657.857327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757657.857332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757657.857336:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757657.857341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757657.857343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757657.857346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757657.857348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757658.049295:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757658.049304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757658.049309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757658.049310:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757658.049360:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757658.049364:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757658.049368:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757658.881318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757658.881327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757658.881329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757658.881334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757658.881338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757658.881343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757658.881345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757658.881349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757658.881351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757659.073283:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757659.073291:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757659.073297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757659.073298:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757659.073358:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757659.073363:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757659.073366:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757659.905304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757659.905314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757659.905316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757659.905323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757659.905328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757659.905333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757659.905336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757659.905340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757659.905344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757660.097280:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757660.097289:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757660.097298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757660.097301:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757660.097400:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757660.097404:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757660.097407:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757660.353292:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757660.353309:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757660.353314:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757660.353320:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757660.353329:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757660.353340:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10715 00000100:00080000:3.0:1787757660.353355:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757660.353359:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757660.353362:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757660.353367:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757660.353371:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757660.353375:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757660.353379:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757660.353382:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757660.353384:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757660.353390:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757660.353393:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757660.353396:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757660.353400:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757660.353404:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353407:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757660.353410:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757660.353421:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757660.353424:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353428:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757660.353431:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757660.353436:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353449:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757660.353461:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659670144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757660.353476:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757660.353483:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353503:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757660.353507:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757660.353512:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757660.353514:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757660.353516:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757660.353520:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757660.353523:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10715 00000100:00080000:3.0:1787757660.353527:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757660.353529:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757660.353531:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757660.353534:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757660.353536:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757660.353560:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757660.353564:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757660.353567:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757660.353570:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757660.353576:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757660.353581:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757660.353584:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757660.353591:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757660.353595:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757660.353596:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353605:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757660.353608:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787757660.353612:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757660.353617:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:3.0:1787757660.353619:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.353627:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757660.353627:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.353638:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757660.353639:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757660.353644:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353649:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757660.353652:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757660.353653:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757660.353660:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757660.353662:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757660.353673:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353676:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757660.353684:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659670144:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787757660.353695:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659670272/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757660.353697:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757660.353705:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757660.353711:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353716:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757660.353718:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757660.353724:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757660.353729:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353732:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757660.353739:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757660.353744:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757660.353746:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757660.353748:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757660.353752:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757660.353753:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757660.353758:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00080000:3.0:1787757660.353763:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10715 00000100:00000001:1.0:1787757660.353767:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757660.353771:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757660.353777:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757660.353779:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757660.353783:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757660.353785:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757660.353788:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757660.353792:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757660.353795:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757660.353796:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757660.353802:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757660.353806:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757660.353808:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757660.353812:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757660.353815:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353817:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757660.353820:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757660.353823:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.353825:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757660.353833:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757660.353834:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757660.353840:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.353843:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305388225 00000100:00000001:3.0:1787757660.353850:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757660.353857:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757660.353861:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757660.353862:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.353873:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305388225 00000100:00000001:3.0:1787757660.353882:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.353884:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757660.353895:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757660.353902:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757660.353905:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659670400/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757660.353909:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.353920:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757660.353934:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757660.353938:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757660.353940:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757660.353947:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757660.353948:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.353951:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757660.353956:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757660.353959:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.353961:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757660.353967:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757660.353971:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757660.353973:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757660.353979:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757660.353981:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757660.353987:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757660.353990:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757660.353994:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757660.353999:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757660.354000:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10715 00000100:00080000:3.0:1787757660.354013:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757660.354014:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670272:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757660.354023:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757660.354026:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757660.354028:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757660.354032:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787757660.354035:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:1.0:1787757660.354037:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757660.354040:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757660.354043:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787757660.354047:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757660.354052:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:1.0:1787757660.354053:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.354055:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757660.354061:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757660.354067:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757660.354069:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757660.354072:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:3.0:1787757660.354077:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354083:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757660.354083:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757660.354092:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757660.354095:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.354098:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757660.354103:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757660.354107:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757660.354108:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.354118:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354119:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305387533 00000100:00000001:1.0:1787757660.354128:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305388225 02000000:00000001:3.0:1787757660.354129:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757660.354135:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354138:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305387533 00000100:00000001:3.0:1787757660.354142:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757660.354145:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305388225 02000000:00000001:3.0:1787757660.354147:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757660.354155:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757660.354156:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354168:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757660.354169:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354177:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757660.354231:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659670528/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757660.354241:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757660.354254:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757660.354263:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:3.0:1787757660.354264:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757660.354269:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757660.354275:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354276:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757660.354281:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757660.354288:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757660.354291:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354292:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757660.354304:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757660.354308:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757660.354312:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757660.354320:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670400:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757660.354329:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757660.354333:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757660.354337:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757660.354341:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757660.354348:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354362:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757660.354364:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757660.354371:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757660.354395:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354402:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305387531 00000100:00000001:1.0:1787757660.354406:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305388225 00000100:00000001:1.0:1787757660.354410:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305387531 00000100:00000001:1.0:1787757660.354416:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305388225 00000100:00000001:1.0:1787757660.354420:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757660.354424:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354429:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354433:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757660.354439:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757660.354442:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757660.354446:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757660.354449:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757660.354451:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757660.354454:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757660.354457:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757660.354462:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757660.354465:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757660.354469:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757660.354477:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757660.354485:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757660.354489:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757660.354492:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757660.354495:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757660.354502:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354512:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757660.354516:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757660.354522:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757660.354572:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354581:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305387531 00000100:00000001:1.0:1787757660.354586:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305388225 00000100:00000001:1.0:1787757660.354590:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305387531 00000100:00000001:1.0:1787757660.354594:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305388225 00000100:00000001:1.0:1787757660.354598:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757660.354602:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354608:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354611:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757660.354617:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757660.354620:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757660.354624:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757660.354626:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757660.354629:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757660.354633:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757660.929364:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757660.929391:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757660.929393:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757660.929399:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757660.929403:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757660.929407:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757660.929409:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757660.929412:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757660.929415:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757661.121377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757661.121386:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757661.121393:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757661.121395:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757661.121622:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757661.121628:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757661.121633:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757661.953303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757661.953312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757661.953315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757661.953321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757661.953325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757661.953330:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757661.953334:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757661.953337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757661.953341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757662.145352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757662.145360:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757662.145368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757662.145374:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757662.145378:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757662.145382:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757662.145386:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757662.145388:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757662.145390:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757662.145393:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757662.145464:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757662.145470:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757662.145475:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757662.977405:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757662.977418:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757662.977421:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757662.977429:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757662.977434:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757662.977440:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757662.977443:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757662.977448:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757662.977452:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757663.169401:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757663.169410:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757663.169419:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757663.169421:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757663.169534:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757663.169545:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757663.169555:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787757663.425567:0:2358:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1787757663.425624:0:2358:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787757663.425645:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757663.425663:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757663.425688:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757663.425692:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757663.425695:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757663.425706:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757663.425712:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757663.425715:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757663.425717:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757663.425719:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757663.425724:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757663.425729:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757663.425732:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757663.425733:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757663.425735:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757663.425740:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757663.425747:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757663.425753:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757663.425756:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757663.425758:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757663.425779:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757663.425789:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757663.425794:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757663.425814:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757660/real 1787757663] req@ffff8ffb45c33800 x1874588659670528/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757663.425840:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757663.425845:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757663.425856:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757663.425872:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757663.425876:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.425897:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757663.425901:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757663.425911:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659670528/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757663.425923:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.425928:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757663.425938:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.425940:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757663.425942:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.425946:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757663.425962:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757663.425967:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757663.425975:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757663.425980:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757663.425989:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757663.425999:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757663.426007:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757660/real 1787757663] req@ffff8ffb45c31180 x1874588659670400/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757663.426026:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757663.426028:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757663.426034:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757663.426038:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757663.426041:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426045:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757663.426047:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757663.426054:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659670400/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757663.426069:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426073:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757663.426078:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426083:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757663.426086:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426089:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757663.426110:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757663.426116:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757663.426123:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757663.426128:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757663.426134:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670400:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757663.426142:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757663.426151:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757660/real 1787757663] req@ffff8ffb45c33480 x1874588659670272/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757663.426169:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757663.426171:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757663.426177:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757663.426234:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757663.426239:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426243:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757663.426245:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757663.426254:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659670272/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757663.426269:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426273:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757663.426278:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426281:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757663.426284:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426287:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757663.426308:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757663.426313:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757663.426321:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757663.426326:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757663.426334:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670272:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757663.426341:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757663.426348:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757660/real 1787757663] req@ffff8ffb45c30000 x1874588659670144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757663.426364:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757663.426367:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757663.426373:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757663.426376:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757663.426379:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426382:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757663.426384:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757663.426393:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659670144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757663.426407:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426413:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757663.426419:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426423:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757663.426426:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426431:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757663.426435:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757663.426440:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757663.426446:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757663.426453:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757663.426461:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659670144:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757663.426470:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426475:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757663.426478:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757663.426565:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757663.426576:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757663.426581:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757663.426587:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757663.426593:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757663.426597:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757663.426602:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757663.426604:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426609:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757663.426611:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757663.426615:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757663.426621:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757663.426623:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757663.426626:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757663.426629:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757663.426632:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757663.426636:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757663.426639:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426644:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757663.426649:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757663.426652:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757663.426659:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757663.426662:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757663.426664:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757663.426670:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757663.426673:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757663.426678:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757663.426681:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426685:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757663.426688:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757663.426691:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757663.426697:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757663.426700:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757663.426702:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757663.426706:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757663.426708:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757663.426711:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757663.426713:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426718:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757663.426731:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757663.426734:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757663.426738:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757663.426742:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757664.001299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757664.001308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757664.001310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757664.001316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757664.001321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757664.001325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757664.001328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757664.001331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757664.001335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757664.193318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757664.193328:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757664.193332:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757664.193336:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757664.193343:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757664.193346:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757664.193350:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757664.193352:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757664.193357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757664.193359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757664.193467:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757664.193526:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757664.193534:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757665.025315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757665.025325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757665.025328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757665.025334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757665.025338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757665.025343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757665.025346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757665.025349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757665.025353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757665.217327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757665.217336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757665.217343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757665.217345:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757665.217508:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757665.217515:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757665.217520:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757665.473307:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757665.473322:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757665.473325:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757665.473330:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757665.473337:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757665.473349:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10720 00000100:00080000:3.0:1787757665.473369:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757665.473376:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757665.473380:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757665.473388:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757665.473406:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757665.473412:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757665.473418:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757665.473424:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757665.473428:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757665.473440:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757665.473475:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757665.473481:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757665.473487:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757665.473492:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473495:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757665.473500:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757665.473514:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757665.473517:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473521:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757665.473525:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757665.473531:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473547:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757665.473561:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659670656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757665.473576:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757665.473583:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473603:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757665.473608:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757665.473614:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757665.473617:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757665.473621:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757665.473626:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757665.473630:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787757665.473633:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10720 00000100:00000001:2.0:1787757665.473640:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787757665.473645:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757665.473648:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757665.473652:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757665.473654:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757665.473659:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757665.473661:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757665.473673:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:2.0:1787757665.473675:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:3.0:1787757665.473681:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787757665.473683:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757665.473692:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757665.473692:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757665.473704:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00100000:2.0:1787757665.473708:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659670656:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757665.473716:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757665.473720:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757665.473723:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787757665.473729:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757665.473736:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757665.473740:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787757665.473744:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757665.473756:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473773:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757665.473780:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757665.473782:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757665.473789:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 02000000:00000001:3.0:1787757665.473792:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757665.473801:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757665.473803:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473811:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757665.473821:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757665.473828:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473833:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757665.473845:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757665.473848:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473853:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757665.473857:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757665.473862:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787757665.473867:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757665.473886:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757665.473891:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757665.473909:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659670784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757665.473911:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757665.473918:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.473927:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757665.473940:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757665.473942:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.473948:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.473956:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305393345 00000100:00000001:3.0:1787757665.473960:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757665.473970:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757665.473978:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787757665.473979:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757665.473987:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757665.473994:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757665.473996:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305393345 00000100:00000001:3.0:1787757665.474003:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757665.474011:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787757665.474014:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10720 00000100:00000001:2.0:1787757665.474024:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757665.474029:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757665.474034:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.474038:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757665.474045:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757665.474046:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757665.474056:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757665.474060:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757665.474068:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00100000:2.0:1787757665.474069:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757665/real 1787757665] req@ffff8ffb45c30000 x1874588659670656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757665.474077:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757665.474088:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757665.474095:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757665.474112:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:2.0:1787757665.474114:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757665.474122:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757665.474123:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757665.474133:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757665.474137:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787757665.474142:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757665.474147:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787757665.474151:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474155:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757665.474160:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757665.474165:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00100000:3.0:1787757665.474169:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757665.474172:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757665.474234:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659670656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:3.0:1787757665.474238:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.474248:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757665.474263:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757665.474269:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.474277:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757665.474283:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757665.474291:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474293:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.474304:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757665.474307:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474312:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.474320:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757665.474327:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757665.474329:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659670912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757665.474333:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757665.474341:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757665.474355:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757665.474367:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00080000:2.0:1787757665.474375:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757665.474379:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474385:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757665.474389:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757665.474399:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757665.474402:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659670656:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757665.474413:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757665.474421:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757665.474422:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757665.474426:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757665.474431:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757665.474438:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757665.474443:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757665.474475:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757665.474480:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757665.474484:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757665.474490:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757665.474493:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757665.474499:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757665.474502:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.474510:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.474523:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757665.474527:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474533:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757665.474538:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757665.474541:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757665.474545:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757665.474549:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.474555:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757665.474560:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757665.474565:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757665.474575:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670784:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757665.474583:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757665.474588:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757665.474591:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.474595:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757665.474601:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474613:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757665.474615:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:2.0:1787757665.474623:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757665.474654:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757665.474665:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757665.474669:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757665.474672:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474676:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757665.474682:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474689:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305392654 00000100:00000001:2.0:1787757665.474695:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305393345 00000100:00000001:2.0:1787757665.474700:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305392654 00000100:00000001:2.0:1787757665.474705:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305393345 00000100:00000001:2.0:1787757665.474709:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757665.474714:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.474718:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757665.474727:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757665/real 1787757665] req@ffff8ffb45c33480 x1874588659670784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757665.474747:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757665.474750:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757665.474756:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757665.474759:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757665.474762:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.474765:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757665.474768:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757665.474777:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659670784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757665.474807:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.474812:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757665.474817:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.474821:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757665.474824:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.474827:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757665.474845:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757665.474857:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757665.474859:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10720 00000100:00080000:2.0:1787757665.474871:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757665.474876:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757665.474885:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00080000:2.0:1787757665.474888:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757665.474891:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757665.474900:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757665.474910:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670784:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757665.474941:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757665.474944:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757665.474947:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.474951:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757665.474955:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757665.474959:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757665.474966:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670912:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757665.474972:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757665.474976:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757665.474979:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.474982:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757665.474989:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475011:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757665.475014:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:2.0:1787757665.475023:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757665.475052:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757665.475062:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757665.475069:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757665.475072:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475076:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757665.475083:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475090:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305392651 00000100:00000001:2.0:1787757665.475096:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305393345 00000100:00000001:2.0:1787757665.475100:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305392651 00000100:00000001:2.0:1787757665.475104:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305393345 00000100:00000001:2.0:1787757665.475109:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757665.475113:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475118:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757665.475127:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757665/real 1787757665] req@ffff8ffb45c31180 x1874588659670912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757665.475145:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757665.475149:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757665.475155:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757665.475158:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757665.475161:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475165:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757665.475167:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757665.475176:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659670912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757665.475234:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757665.475241:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757665.475247:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757665.475250:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757665.475257:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787757665.475260:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757665.475272:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757665.475274:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757665.475280:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757665.475290:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757665.475297:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475301:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757665.475321:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757665.475327:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757665.475336:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757665.475341:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757665.475348:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659670912:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757665.475358:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475362:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757665.475365:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757665.475368:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757665.475375:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757665.475378:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00080000:3.0:1787757665.475381:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000020:00000001:2.0:1787757665.475383:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757665.475395:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787757665.475395:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.475402:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 02000000:00000001:3.0:1787757665.475405:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757665.475412:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 02000000:00000001:3.0:1787757665.475417:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757665.475419:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757665.475427:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475429:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757665.475437:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757665.475440:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757665.475469:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757665.475475:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000008:00100000:3.0:1787757665.475477:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000020:00000001:2.0:1787757665.475479:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757665.475484:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000008:00000001:3.0:1787757665.475489:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.475493:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787757665.475497:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.475500:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757665.475507:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757665.475510:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757665.475513:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757665.475516:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757665.475520:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475529:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757665.475530:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757665.475535:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475541:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757665.475546:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757665.475554:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757665.475570:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.475585:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757665.475600:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659671040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757665.475619:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757665.475626:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757665.475649:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475687:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757665.475694:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757665.475698:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757665.475701:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757665.475704:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.475712:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757665.475715:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757665.475717:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757665.475724:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757665.475729:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757665.475732:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757665.475735:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757665.475737:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757665.475742:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475751:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757665.475753:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:2.0:1787757665.475758:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757665.475777:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757665.475785:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757665.475789:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757665.475791:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475793:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757665.475797:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475801:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305392652 00000100:00000001:2.0:1787757665.475805:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305393345 00000100:00000001:2.0:1787757665.475808:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305392652 00000100:00000001:2.0:1787757665.475811:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305393345 00000100:00000001:2.0:1787757665.475814:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757665.475816:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475818:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757665.475823:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757665/real 1787757665] req@ffff8ffb45c33800 x1874588659671040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757665.475834:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757665.475835:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757665.475838:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757665.475840:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757665.475842:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475844:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757665.475846:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757665.475851:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659671040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757665.475860:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475862:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757665.475866:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475868:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757665.475870:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475872:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757665.475885:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757665.475888:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757665.475893:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757665.475895:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757665.475900:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757665.475904:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475906:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757665.475907:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757665.475909:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757665.475912:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757665.475913:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757665.475915:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757665.475917:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757665.475919:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757665.475921:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757665.475923:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475925:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757665.475930:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757665.475932:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757665.475934:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757665.475936:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757666.049333:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757666.049346:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757666.049350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757666.049360:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757666.049368:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757666.049373:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757666.049376:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757666.049380:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757666.049383:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757666.241303:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757666.241312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757666.241322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757666.241325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757666.241378:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757666.241385:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757666.241391:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757667.073344:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757667.073352:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757667.073355:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757667.073362:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757667.073366:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757667.073370:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757667.073374:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757667.073377:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757667.073382:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757667.265334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757667.265342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757667.265351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757667.265353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757667.265518:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757667.265524:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757667.265529:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757668.097376:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757668.097389:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757668.097393:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757668.097422:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757668.097429:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757668.097435:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757668.097439:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757668.097443:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757668.097448:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757668.289381:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757668.289393:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757668.289434:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757668.289436:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757668.289554:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757668.289566:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757668.289575:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757669.121340:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757669.121352:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757669.121356:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757669.121364:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757669.121371:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757669.121377:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757669.121382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757669.121414:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757669.121420:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757669.313334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757669.313342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757669.313349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757669.313351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757669.313446:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757669.313452:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757669.313457:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757670.145275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757670.145284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757670.145287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757670.145294:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757670.145298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757670.145303:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757670.145306:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757670.145310:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757670.145313:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757670.337318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757670.337328:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757670.337338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757670.337341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757670.337469:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757670.337478:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757670.337485:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757670.593275:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757670.593286:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757670.593290:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757670.593295:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757670.593301:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757670.593310:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10725 00000100:00080000:3.0:1787757670.593323:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757670.593328:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757670.593331:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757670.593335:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757670.593339:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757670.593342:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757670.593345:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757670.593349:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757670.593350:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757670.593357:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757670.593399:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757670.593402:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757670.593407:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757670.593411:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.593414:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757670.593418:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757670.593432:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757670.593435:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.593439:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757670.593442:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757670.593448:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.593463:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757670.593476:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659671168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757670.593490:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757670.593496:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.593517:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757670.593521:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757670.593526:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757670.593529:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757670.593531:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757670.593535:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757670.593540:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10725 00000100:00080000:3.0:1787757670.593546:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757670.593550:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757670.593553:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757670.593557:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757670.593559:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757670.593562:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757670.593565:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757670.593567:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757670.593570:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757670.593570:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757670.593579:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787757670.593581:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757670.593587:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757670.593588:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593594:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757670.593595:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757670.593601:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757670.593606:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757670.593609:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757670.593613:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.593616:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757670.593619:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:2.0:1787757670.593623:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757670.593628:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757670.593636:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659671168:192.168.201.118@tcp:250:kworker.0 00000008:00000001:3.0:1787757670.593637:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.593644:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593650:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757670.593654:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757670.593658:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.593659:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757670.593661:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757670.593664:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757670.593666:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757670.593672:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757670.593674:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.593682:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.593685:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757670.593697:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659671296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757670.593707:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757670.593709:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757670.593712:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757670.593716:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593722:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757670.593726:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757670.593732:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757670.593736:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757670.593739:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757670.593743:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757670.593747:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757670.593751:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10725 00000100:00080000:3.0:1787757670.593766:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757670.593770:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757670.593771:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757670.593774:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757670.593776:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757670.593778:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757670.593781:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757670.593784:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757670.593789:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:2.0:1787757670.593791:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757670.593799:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757670.593803:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757670.593807:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757670.593812:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757670.593814:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757670.593820:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757670.593823:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593826:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.593835:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757670.593836:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:3.0:1787757670.593846:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757670.593849:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593859:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305398465 00000008:00000001:3.0:1787757670.593861:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.593870:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593877:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757670.593880:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757670.593887:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593888:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305398465 00000100:00000001:3.0:1787757670.593896:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787757670.593900:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757670.593903:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757670.593907:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757670.593912:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593914:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593924:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757670.593932:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757670.593936:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757670/real 1787757670] req@ffff8ffb45c32a00 x1874588659671168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757670.593948:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659671424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:2.0:1787757670.593967:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757670.593973:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757670.593975:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757670.593980:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757670.593982:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593987:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757670.593992:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.593993:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594004:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757670.594007:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757670.594008:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757670.594021:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659671168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757670.594024:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757670.594033:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757670.594041:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757670.594042:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594050:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757670.594054:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757670.594056:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594063:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757670.594067:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757670.594070:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10725 00000100:00000001:2.0:1787757670.594076:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757670.594084:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757670.594086:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757670.594095:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00080000:2.0:1787757670.594097:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757670.594098:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757670.594110:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757670.594113:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757670.594118:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:2.0:1787757670.594119:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000020:00000001:3.0:1787757670.594126:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757670.594132:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757670.594133:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757670.594136:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757670.594152:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1787757670.594153:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659671168:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757670.594156:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757670.594164:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757670.594171:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757670.594176:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757670.594178:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594184:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757670.594221:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757670.594225:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757670.594229:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.594232:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757670.594239:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757670.594242:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594246:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757670.594247:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757670.594252:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757670.594253:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757670.594260:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.594261:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757670.594268:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757670.594271:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757670.594272:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.594279:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757670.594285:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00100000:3.0:1787757670.594287:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659671552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757670.594291:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757670.594296:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594307:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757670.594318:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757670.594323:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757670.594324:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594330:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757670.594334:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594336:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757670.594342:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757670.594345:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757670.594348:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757670.594350:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.594354:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757670.594356:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757670.594387:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757670.594394:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671296:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757670.594401:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757670.594406:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757670.594409:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.594413:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757670.594419:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594432:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757670.594435:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787757670.594443:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757670.594471:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757670.594482:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757670.594487:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757670.594489:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594493:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757670.594499:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594507:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305397774 00000100:00000001:2.0:1787757670.594511:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305398465 00000100:00000001:2.0:1787757670.594514:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305397774 00000100:00000001:2.0:1787757670.594517:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305398465 00000100:00000001:2.0:1787757670.594520:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757670.594523:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594525:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757670.594532:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757670/real 1787757670] req@ffff8ffb45c30700 x1874588659671296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757670.594546:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757670.594548:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757670.594552:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757670.594554:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757670.594556:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594558:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757670.594560:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757670.594566:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659671296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757670.594575:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594578:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757670.594581:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594583:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757670.594585:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594587:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757670.594602:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757670.594605:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.594609:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757670.594612:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757670.594616:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671296:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757670.594621:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757670.594623:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757670.594625:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.594628:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757670.594630:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757670.594633:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757670.594636:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671424:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757670.594640:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757670.594643:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757670.594645:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.594648:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757670.594652:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594659:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757670.594661:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:2.0:1787757670.594666:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757670.594683:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757670.594688:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757670.594691:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757670.594693:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594695:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757670.594699:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594702:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305397771 00000100:00000001:2.0:1787757670.594706:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305398465 00000100:00000001:2.0:1787757670.594709:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305397771 00000100:00000001:2.0:1787757670.594711:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305398465 00000100:00000001:2.0:1787757670.594714:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757670.594717:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594719:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757670.594724:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757670/real 1787757670] req@ffff8ffb45c31500 x1874588659671424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757670.594733:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757670.594735:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757670.594737:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757670.594739:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757670.594741:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594743:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757670.594744:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757670.594749:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659671424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757670.594757:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594760:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757670.594763:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594765:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757670.594767:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594769:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757670.594782:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757670.594785:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.594789:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757670.594792:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757670.594796:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671424:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757670.594800:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757670.594802:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757670.594803:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.594806:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757670.594808:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757670.594810:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757670.594813:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757670.594817:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757670.594818:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757670.594820:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757670.594822:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757670.594826:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594831:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757670.594833:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:2.0:1787757670.594837:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757670.594849:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757670.594853:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757670.594855:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757670.594858:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594860:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757670.594863:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594867:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305397771 00000100:00000001:2.0:1787757670.594870:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305398465 00000100:00000001:2.0:1787757670.594873:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305397771 00000100:00000001:2.0:1787757670.594876:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305398465 00000100:00000001:2.0:1787757670.594879:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757670.594881:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.594884:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757670.594888:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757670/real 1787757670] req@ffff8ffb45c30e00 x1874588659671552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757670.594897:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757670.594898:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757670.594901:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757670.594903:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757670.594904:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594906:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757670.594908:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757670.594912:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659671552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757670.594921:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594923:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757670.594927:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594929:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757670.594931:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594933:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757670.594945:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757670.594948:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.594951:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757670.594954:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757670.594958:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757670.594963:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594965:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757670.594967:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757670.594970:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757670.594973:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757670.594975:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757670.594976:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757670.594979:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757670.594981:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757670.594984:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757670.594985:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.594987:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757670.594989:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757670.594991:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757670.594993:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757670.594994:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757670.594996:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757670.594998:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757670.594999:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757670.595001:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757670.595003:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.595004:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757670.595006:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757670.595007:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757670.595010:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757670.595011:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757670.595013:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757670.595015:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757670.595016:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757670.595018:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757670.595019:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.595021:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757670.595026:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757670.595028:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757670.595030:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757670.595032:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757671.169293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757671.169302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757671.169305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757671.169311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757671.169316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757671.169321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757671.169324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757671.169327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757671.169330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757671.361310:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757671.361322:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757671.361334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757671.361338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757671.361429:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757671.361436:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757671.361441:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757672.193293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757672.193302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757672.193305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757672.193312:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757672.193316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757672.193321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757672.193324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757672.193327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757672.193354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757672.385367:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757672.385375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757672.385383:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757672.385387:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757672.385391:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757672.385395:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757672.385399:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757672.385402:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757672.385405:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757672.385408:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757672.385498:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757672.385504:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757672.385509:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757673.217306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757673.217334:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757673.217338:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757673.217346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757673.217352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757673.217358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757673.217363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757673.217369:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757673.217374:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757673.409344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757673.409352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757673.409360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757673.409361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757673.409442:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757673.409448:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757673.409454:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757674.241340:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757674.241351:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757674.241353:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757674.241360:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757674.241364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757674.241368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757674.241371:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757674.241374:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757674.241378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757674.433353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757674.433366:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757674.433371:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757674.433376:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757674.433396:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757674.433400:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757674.433403:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757674.433406:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757674.433412:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757674.433414:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757674.433503:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757674.433510:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757674.433515:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757675.265325:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757675.265334:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757675.265337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757675.265344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757675.265349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757675.265353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757675.265357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757675.265360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757675.265364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757675.457408:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757675.457419:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757675.457428:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757675.457430:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757675.457520:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757675.457528:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757675.457534:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757675.713426:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757675.713440:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787757675.713444:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757675.713450:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757675.713457:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757675.713470:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10730 00000100:00080000:0.0:1787757675.713490:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757675.713497:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757675.713502:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757675.713510:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757675.713515:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757675.713520:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787757675.713526:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787757675.713530:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757675.713532:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757675.713542:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757675.713548:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757675.713554:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757675.713561:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757675.713568:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.713573:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757675.713577:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757675.713601:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757675.713605:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.713612:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757675.713617:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757675.713628:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.713657:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757675.713679:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790c000 x1874588659671680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757675.713716:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757675.713724:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.713750:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757675.713755:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757675.713761:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787757675.713763:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757675.713765:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757675.713770:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757675.713773:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10730 00000100:00080000:0.0:1787757675.713780:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757675.713783:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757675.713786:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757675.713791:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757675.713793:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757675.713797:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787757675.713802:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757675.713805:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757675.713807:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757675.713812:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757675.713815:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.713818:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:0.0:1787757675.713819:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757675.713832:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:0.0:1787757675.713836:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757675.713838:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:0.0:1787757675.713852:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.713854:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757675.713857:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757675.713859:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:0.0:1787757675.713867:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787757675.713870:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:0.0:1787757675.713883:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.713885:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757675.713894:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.713897:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757675.713906:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757675.713916:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757675.713917:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659671680:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757675.713923:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757675.713929:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757675.713936:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.713940:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:0.0:1787757675.713948:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.713953:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:0.0:1787757675.713965:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790dc00 x1874588659671808/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757675.713966:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.713982:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:0.0:1787757675.713994:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757675.713997:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.714001:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714005:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757675.714018:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757675.714025:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757675.714028:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757675.714030:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757675.714035:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757675.714039:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10730 00000100:00000001:2.0:1787757675.714042:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:0.0:1787757675.714045:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757675.714048:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133072896 : -123165576478720 : ffff8ffb4790c000) 00000100:00000001:0.0:1787757675.714050:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757675.714059:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757675.714064:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757675.714064:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:0.0:1787757675.714074:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757675.714079:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787757675.714083:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757675.714086:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757675.714088:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757675.714096:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757675.714101:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757675.714103:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757675.714108:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757675.714114:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714117:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757675.714121:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757675.714124:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714126:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757675.714132:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757675.714137:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787757675.714144:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757675.714145:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757675.714153:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757675.714157:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714165:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757675.714173:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f480 x1874588659671936/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757675.714175:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757675.714247:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757675.714250:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714254:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714259:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:0.0:1787757675.714260:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757675.714296:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757675.714299:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757675.714307:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757675.714310:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:0.0:1787757675.714313:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757675.714321:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757675.714326:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757675.714327:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10730 00000100:00080000:0.0:1787757675.714345:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757675.714346:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305403586 00000100:00000001:0.0:1787757675.714365:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757675.714370:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757675.714375:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757675.714377:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757675.714380:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:0.0:1787757675.714381:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:0.0:1787757675.714393:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757675.714397:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757675.714399:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757675.714402:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305403586 00000100:00080000:0.0:1787757675.714405:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757675.714421:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757675.714424:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757675.714427:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:0.0:1787757675.714430:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757675.714441:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714447:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757675.714448:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00100000:0.0:1787757675.714452:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757675.714468:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714470:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714472:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714481:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:0.0:1787757675.714485:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757675.714490:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714495:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757675.714502:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:2.0:1787757675.714505:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757675/real 1787757675] req@ffff8ffb4790c000 x1874588659671680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:0.0:1787757675.714507:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757675.714522:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757675.714547:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ca80 x1874588659672064/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:2.0:1787757675.714552:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757675.714561:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757675.714574:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757675.714574:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757675.714588:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757675.714590:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714599:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757675.714601:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714611:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757675.714618:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757675.714629:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c000 x1874588659671680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757675.714641:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714646:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757675.714651:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714655:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757675.714658:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714661:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757675.714664:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757675.714669:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757675.714675:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757675.714678:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757675.714685:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659671680:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757675.714695:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714699:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757675.714702:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757675.714706:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757675.714714:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757675.714718:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757675.714722:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757675.714730:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757675.714734:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757675.714740:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757675.714743:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714748:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714760:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757675.714765:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714770:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757675.714774:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757675.714777:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757675.714780:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757675.714783:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.714786:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757675.714790:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757675.714793:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757675.714799:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671808:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757675.714804:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757675.714807:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757675.714809:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.714812:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757675.714816:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714826:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757675.714828:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133080064 : -123165576471552 : ffff8ffb4790dc00) 00000100:00000001:2.0:1787757675.714834:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757675.714857:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757675.714864:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757675.714867:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757675.714869:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714871:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757675.714875:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714881:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305402894 00000100:00000001:2.0:1787757675.714885:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305403586 00000100:00000001:2.0:1787757675.714888:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305402894 00000100:00000001:2.0:1787757675.714891:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305403586 00000100:00000001:2.0:1787757675.714894:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757675.714896:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.714899:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757675.714906:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757675/real 1787757675] req@ffff8ffb4790dc00 x1874588659671808/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757675.714917:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757675.714919:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757675.714922:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757675.714925:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757675.714926:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714929:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757675.714931:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757675.714937:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790dc00 x1874588659671808/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757675.714951:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714956:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757675.714961:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714964:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757675.714966:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.714969:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757675.714992:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757675.714998:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757675.715006:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757675.715011:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757675.715018:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671808:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757675.715026:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757675.715030:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757675.715032:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.715037:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757675.715041:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757675.715044:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757675.715048:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757675.715056:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757675.715059:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757675.715062:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.715064:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757675.715069:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715078:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757675.715080:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:2.0:1787757675.715087:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757675.715113:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757675.715122:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757675.715126:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757675.715129:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715132:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757675.715137:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715144:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305402891 00000100:00000001:2.0:1787757675.715150:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305403586 00000100:00000001:2.0:1787757675.715155:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305402891 00000100:00000001:2.0:1787757675.715160:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305403586 00000100:00000001:2.0:1787757675.715165:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757675.715169:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715172:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757675.715182:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757675/real 1787757675] req@ffff8ffb4790f480 x1874588659671936/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757675.715243:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757675.715246:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757675.715253:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757675.715257:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757675.715261:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715265:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757675.715295:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757675.715305:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659671936/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757675.715323:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715327:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757675.715330:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715332:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757675.715334:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715336:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757675.715349:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757675.715352:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757675.715356:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757675.715359:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757675.715364:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659671936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757675.715369:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757675.715372:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757675.715375:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.715380:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757675.715384:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757675.715388:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757675.715394:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757675.715400:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757675.715404:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757675.715407:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757675.715411:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757675.715417:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715430:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757675.715433:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:2.0:1787757675.715440:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757675.715466:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757675.715476:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757675.715481:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757675.715483:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715486:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757675.715490:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715495:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305402892 00000100:00000001:2.0:1787757675.715499:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305403586 00000100:00000001:2.0:1787757675.715502:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305402892 00000100:00000001:2.0:1787757675.715505:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305403586 00000100:00000001:2.0:1787757675.715507:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757675.715510:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715512:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757675.715518:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757675/real 1787757675] req@ffff8ffb4790ca80 x1874588659672064/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757675.715528:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757675.715530:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757675.715533:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757675.715535:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757675.715536:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715539:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757675.715541:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757675.715545:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659672064/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757675.715553:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715556:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757675.715559:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715561:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757675.715562:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715565:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757675.715577:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757675.715580:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757675.715584:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757675.715586:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757675.715591:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757675.715595:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715598:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757675.715600:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757675.715602:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757675.715605:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757675.715607:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757675.715609:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757675.715612:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757675.715614:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757675.715617:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757675.715619:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715621:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757675.715622:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757675.715624:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757675.715627:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757675.715629:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757675.715630:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757675.715632:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757675.715633:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757675.715635:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757675.715637:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715638:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757675.715640:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757675.715641:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757675.715643:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757675.715644:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757675.715645:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757675.715648:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757675.715649:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757675.715651:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757675.715652:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715654:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757675.715659:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757675.715661:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757675.715664:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757675.715666:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757676.289408:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757676.289420:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757676.289424:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757676.289432:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757676.289437:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757676.289443:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757676.289446:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757676.289450:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757676.289454:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757676.481367:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757676.481380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757676.481394:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757676.481398:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757676.481433:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757676.481441:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757676.481448:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757677.313479:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757677.313492:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757677.313496:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757677.313505:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757677.313512:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757677.313519:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757677.313524:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757677.313528:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757677.313532:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757677.505409:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757677.505420:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757677.505428:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757677.505430:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757677.505520:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757677.505528:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757677.505535:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757678.337339:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757678.337349:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757678.337352:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757678.337358:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757678.337363:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757678.337367:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757678.337370:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757678.337373:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757678.337376:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757678.529338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757678.529351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757678.529360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757678.529362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757678.529469:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757678.529475:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757678.529479:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757679.361407:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757679.361423:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757679.361428:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757679.361438:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757679.361446:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757679.361454:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757679.361460:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757679.361464:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757679.361468:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757679.553328:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757679.553337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757679.553346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757679.553349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757679.553441:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757679.553448:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757679.553453:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757680.385291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757680.385300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757680.385303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757680.385309:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757680.385314:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757680.385319:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757680.385322:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757680.385325:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757680.385329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757680.577384:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757680.577394:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757680.577402:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757680.577404:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757680.577498:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757680.577509:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757680.577517:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757680.833419:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757680.833436:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757680.833442:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757680.833448:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757680.833455:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757680.833466:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10735 00000100:00080000:1.0:1787757680.833484:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757680.833491:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757680.833495:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757680.833502:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757680.833507:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757680.833513:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757680.833517:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757680.833522:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757680.833525:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757680.833532:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757680.833536:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757680.833539:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757680.833544:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757680.833548:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.833551:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757680.833555:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757680.833569:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757680.833571:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.833575:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757680.833578:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757680.833583:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.833598:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757680.833610:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659672192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757680.833625:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757680.833633:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.833664:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757680.833670:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757680.833682:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757680.833685:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757680.833688:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757680.833695:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757680.833703:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10735 00000100:00080000:1.0:1787757680.833712:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757680.833717:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757680.833721:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757680.833725:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:1.0:1787757680.833727:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757680.833735:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757680.833738:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:1.0:1787757680.833740:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757680.833747:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:1.0:1787757680.833748:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757680.833757:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757680.833758:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757680.833760:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757680.833768:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757680.833770:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757680.833779:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.833780:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757680.833783:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757680.833788:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757680.833793:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757680.833795:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757680.833806:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.833811:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757680.833818:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757680.833822:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.833828:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757680.833829:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659672192:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757680.833835:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757680.833839:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757680.833841:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.833847:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757680.833848:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757680.833850:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757680.833855:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757680.833858:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.833862:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757680.833869:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.833873:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757680.833883:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659672320/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757680.833897:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757680.833903:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787757680.833913:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757680.833916:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757680.833922:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.833927:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757680.833932:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757680.833936:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757680.833939:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757680.833942:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757680.833946:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757680.833951:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10735 00000100:00080000:1.0:1787757680.833957:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757680.833960:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757680.833962:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:2.0:1787757680.833969:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:1.0:1787757680.833971:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757680.833978:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757680.833981:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757680.833986:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757680.833990:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757680.833990:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757680.833998:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757680.834002:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787757680.834005:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757680.834008:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.834012:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834014:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787757680.834021:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757680.834027:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757680.834029:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757680.834037:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834037:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305408705 00000100:00000001:1.0:1787757680.834041:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757680.834048:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757680.834052:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.834055:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834057:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:1.0:1787757680.834063:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757680.834068:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305408705 02000000:00000001:1.0:1787757680.834069:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.834078:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757680.834080:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757680.834083:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787757680.834084:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.834098:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757680.834104:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659672448/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757680.834108:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757680.834119:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834129:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757680.834138:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757680/real 1787757680] req@ffff8ffb4a537800 x1874588659672192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757680.834144:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757680.834152:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757680.834155:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757680.834159:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757680.834170:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757680.834176:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000400:00000001:2.0:1787757680.834266:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757680.834269:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757680.834275:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757680.834278:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757680.834282:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.834286:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:1.0:1787757680.834295:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757680.834295:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757680.834309:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:2.0:1787757680.834311:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659672192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757680.834327:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10735 00000100:00000001:2.0:1787757680.834341:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757680.834346:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757680.834349:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757680.834353:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757680.834360:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757680.834361:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757680.834368:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:1.0:1787757680.834370:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757680.834373:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787757680.834376:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757680.834380:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000020:00000001:1.0:1787757680.834382:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757680.834386:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:1.0:1787757680.834388:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:2.0:1787757680.834393:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787757680.834396:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757680.834402:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757680.834408:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1787757680.834414:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757680.834422:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834422:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:1.0:1787757680.834434:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:2.0:1787757680.834445:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659672192:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757680.834446:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757680.834458:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834464:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757680.834467:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757680.834476:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00100000:1.0:1787757680.834478:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757680.834486:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000008:00000001:1.0:1787757680.834489:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.834494:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:1.0:1787757680.834497:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.834505:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:1.0:1787757680.834512:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000020:00000001:2.0:1787757680.834513:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:1.0:1787757680.834523:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757680.834524:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:1.0:1787757680.834531:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757680.834534:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757680.834536:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757680.834540:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 02000000:00000001:1.0:1787757680.834543:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834546:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757680.834551:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757680.834555:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834556:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.834575:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00100000:1.0:1787757680.834576:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659672576/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757680.834580:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834590:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757680.834595:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757680.834602:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757680.834603:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757680.834607:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757680.834612:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834614:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.834624:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757680.834627:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.834628:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757680.834639:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757680.834648:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672320:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757680.834653:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757680.834656:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757680.834659:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.834662:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757680.834670:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834685:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757680.834688:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787757680.834698:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757680.834727:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757680.834737:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757680.834743:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757680.834745:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834749:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757680.834755:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834763:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305408014 00000100:00000001:2.0:1787757680.834769:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305408705 00000100:00000001:2.0:1787757680.834772:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305408014 00000100:00000001:2.0:1787757680.834776:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305408705 00000100:00000001:2.0:1787757680.834779:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757680.834782:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834784:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757680.834791:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757680/real 1787757680] req@ffff8ffb4a536300 x1874588659672320/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757680.834804:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757680.834805:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757680.834809:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757680.834811:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757680.834813:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.834816:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757680.834817:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757680.834822:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659672320/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757680.834832:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.834835:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757680.834838:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.834840:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757680.834841:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.834843:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757680.834859:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757680.834862:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757680.834866:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757680.834869:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757680.834873:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672320:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757680.834878:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757680.834879:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757680.834881:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.834884:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757680.834887:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757680.834890:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757680.834894:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672448:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757680.834898:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757680.834901:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757680.834903:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.834905:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757680.834909:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834916:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757680.834918:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787757680.834923:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757680.834939:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757680.834945:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757680.834948:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757680.834950:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834952:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757680.834955:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834959:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305408011 00000100:00000001:2.0:1787757680.834963:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305408705 00000100:00000001:2.0:1787757680.834966:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305408011 00000100:00000001:2.0:1787757680.834968:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305408705 00000100:00000001:2.0:1787757680.834971:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757680.834974:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.834976:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757680.834981:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757680/real 1787757680] req@ffff8ffb4a534a80 x1874588659672448/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757680.834990:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757680.834991:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757680.834994:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757680.834996:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757680.834998:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835000:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757680.835001:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757680.835006:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659672448/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757680.835016:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835018:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757680.835021:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835023:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757680.835024:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835026:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757680.835038:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757680.835041:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757680.835044:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757680.835047:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757680.835051:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672448:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757680.835055:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835057:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757680.835059:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757680.835061:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757680.835064:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757680.835066:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757680.835068:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757680.835072:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757680.835074:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757680.835076:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757680.835078:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835080:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757680.835082:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757680.835083:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757680.835086:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757680.835087:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757680.835088:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757680.835090:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757680.835092:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757680.835094:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757680.835095:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835097:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835102:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757680.835104:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.835107:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757680.835109:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757680.835111:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757680.835113:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757680.835114:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.835117:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757680.835119:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757680.835121:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757680.835125:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757680.835129:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757680.835131:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757680.835133:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757680.835135:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757680.835139:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.835144:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757680.835145:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757680.835150:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757680.835162:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757680.835167:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757680.835169:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757680.835171:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.835173:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757680.835176:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.835238:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305408012 00000100:00000001:2.0:1787757680.835242:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305408705 00000100:00000001:2.0:1787757680.835245:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305408012 00000100:00000001:2.0:1787757680.835248:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305408705 00000100:00000001:2.0:1787757680.835251:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757680.835253:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.835256:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757680.835262:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757680/real 1787757680] req@ffff8ffb4a534700 x1874588659672576/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757680.835282:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757680.835284:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757680.835288:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757680.835291:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757680.835292:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835294:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757680.835297:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757680.835301:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659672576/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757680.835310:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835313:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757680.835316:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835318:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757680.835319:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835322:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757680.835334:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757680.835337:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757680.835340:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757680.835343:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757680.835346:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757680.835351:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835354:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757680.835356:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757680.835358:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757680.835360:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757680.835362:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757680.835364:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757680.835367:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757680.835368:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757680.835371:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757680.835372:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835375:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757680.835379:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757680.835381:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757680.835384:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757680.835386:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757681.409389:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757681.409402:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757681.409405:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757681.409414:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757681.409419:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757681.409426:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757681.409429:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757681.409433:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757681.409437:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757681.601334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757681.601342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757681.601351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757681.601354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757681.601381:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757681.601390:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757681.601399:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757682.433321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757682.433331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757682.433333:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757682.433340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757682.433345:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757682.433350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757682.433353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757682.433357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757682.433360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757682.625329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757682.625336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757682.625344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757682.625348:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757682.625352:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757682.625357:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757682.625362:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757682.625365:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757682.625368:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757682.625371:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757682.625488:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757682.625504:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757682.625512:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757683.457379:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757683.457395:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757683.457400:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757683.457411:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757683.457419:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757683.457427:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757683.457432:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757683.457438:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757683.457444:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757683.649321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757683.649330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757683.649341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757683.649344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757683.649486:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757683.649493:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757683.649499:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757684.481292:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757684.481301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757684.481304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757684.481311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757684.481315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757684.481320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757684.481323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757684.481326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757684.481330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757684.673302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757684.673312:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757684.673317:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757684.673321:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757684.673327:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757684.673330:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757684.673333:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757684.673337:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757684.673342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757684.673344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757684.673494:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757684.673500:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757684.673505:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757685.505444:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757685.505457:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757685.505462:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757685.505471:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757685.505476:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757685.505482:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757685.505486:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757685.505490:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757685.505494:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757685.697293:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757685.697302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757685.697311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757685.697313:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757685.697461:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757685.697470:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757685.697477:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757685.953315:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757685.953332:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757685.953338:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757685.953345:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757685.953354:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757685.953368:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10740 00000100:00080000:1.0:1787757685.953389:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757685.953397:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757685.953402:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757685.953411:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757685.953417:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757685.953423:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757685.953429:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757685.953433:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757685.953436:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757685.953443:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757685.953447:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757685.953452:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757685.953458:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757685.953463:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953466:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757685.953470:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757685.953489:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757685.953491:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953497:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757685.953501:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757685.953508:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953527:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757685.953542:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659672704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757685.953559:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757685.953566:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953590:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757685.953595:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757685.953600:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757685.953602:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757685.953605:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757685.953609:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757685.953612:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10740 00000100:00080000:1.0:1787757685.953618:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757685.953621:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757685.953623:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757685.953626:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757685.953628:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757685.953631:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757685.953634:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757685.953637:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757685.953638:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757685.953642:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757685.953645:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757685.953648:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757685.953657:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757685.953660:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953662:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757685.953669:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757685.953673:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953677:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757685.953683:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757685.953686:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953689:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757685.953691:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757685.953694:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953700:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757685.953708:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659672832/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757685.953710:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757685.953720:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757685.953724:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757685.953726:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757685.953734:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.953735:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757685.953741:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953751:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757685.953751:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757685.953758:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757685.953762:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787757685.953765:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757685.953780:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787757685.953782:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659672704:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757685.953791:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757685.953796:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757685.953800:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787757685.953806:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757685.953811:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:2.0:1787757685.953813:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757685.953821:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:1.0:1787757685.953827:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10740 02000000:00000001:2.0:1787757685.953831:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757685.953839:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757685.953846:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757685.953849:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757685.953853:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757685.953855:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757685.953858:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757685.953862:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757685.953863:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757685.953866:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757685.953869:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:1.0:1787757685.953874:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757685.953879:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787757685.953885:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757685.953891:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757685.953893:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757685.953898:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757685.953901:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953904:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757685.953908:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757685.953911:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953914:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757685.953922:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757685.953924:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953927:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757685.953929:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757685.953933:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953943:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757685.953949:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659672960/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757685.953963:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757685.953967:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757685.953973:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.953977:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757685.953982:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757685.953987:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757685.953990:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757685.953992:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787757685.953993:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757685.954004:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757685.954011:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787757685.954012:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10740 00000100:00000001:2.0:1787757685.954023:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757685.954024:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757685.954030:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757685.954037:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757685.954038:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757685.954046:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757685.954050:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757685.954053:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757685.954057:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757685.954058:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.954061:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757685.954068:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757685.954072:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305413825 00000100:00080000:1.0:1787757685.954074:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757685.954084:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757685.954087:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757685.954120:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757685.954125:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.954128:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757685.954128:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00100000:1.0:1787757685.954134:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757685.954140:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.954143:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954149:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305413825 02000000:00000001:1.0:1787757685.954151:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757685.954156:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.954162:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757685.954164:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757685.954167:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954169:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757685.954178:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954182:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787757685.954223:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659673088/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757685.954236:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757685.954239:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757685.954243:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954261:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954266:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757685.954283:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757685/real 1787757685] req@ffff8ffb4a535c00 x1874588659672704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757685.954303:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757685.954306:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757685.954313:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757685.954316:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757685.954319:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954322:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757685.954325:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757685.954332:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659672704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757685.954342:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954347:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757685.954352:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954354:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757685.954356:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954359:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757685.954362:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757685.954366:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757685.954371:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757685.954374:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757685.954380:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659672704:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757685.954388:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954391:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757685.954394:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757685.954398:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757685.954403:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757685.954406:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757685.954409:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757685.954417:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757685.954420:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757685.954423:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757685.954425:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954428:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954439:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757685.954442:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954445:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757685.954448:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757685.954450:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757685.954452:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757685.954454:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757685.954458:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757685.954460:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757685.954463:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757685.954468:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672832:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757685.954472:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757685.954475:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757685.954477:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757685.954479:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757685.954484:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954492:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757685.954494:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787757685.954501:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757685.954521:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757685.954527:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757685.954531:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757685.954532:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954535:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757685.954539:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954545:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305413134 00000100:00000001:2.0:1787757685.954549:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305413825 00000100:00000001:2.0:1787757685.954553:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305413134 00000100:00000001:2.0:1787757685.954555:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305413825 00000100:00000001:2.0:1787757685.954558:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757685.954561:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954563:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757685.954568:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757685/real 1787757685] req@ffff8ffb4a537b80 x1874588659672832/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757685.954578:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757685.954580:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757685.954582:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757685.954585:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757685.954586:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954589:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757685.954590:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757685.954595:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659672832/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757685.954604:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954606:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757685.954609:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954611:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757685.954612:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954614:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757685.954636:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757685.954640:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757685.954644:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757685.954647:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757685.954652:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672832:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757685.954657:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757685.954659:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757685.954661:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757685.954663:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757685.954665:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757685.954668:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757685.954671:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672960:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757685.954675:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757685.954677:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757685.954679:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757685.954681:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757685.954685:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954691:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757685.954693:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787757685.954699:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757685.954711:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757685.954716:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757685.954718:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757685.954721:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954723:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757685.954726:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954730:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305413131 00000100:00000001:2.0:1787757685.954734:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305413825 00000100:00000001:2.0:1787757685.954736:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305413131 00000100:00000001:2.0:1787757685.954739:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305413825 00000100:00000001:2.0:1787757685.954742:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757685.954745:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954747:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757685.954751:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757685/real 1787757685] req@ffff8ffb4a534700 x1874588659672960/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757685.954769:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757685.954771:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757685.954774:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757685.954776:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757685.954778:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954780:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757685.954781:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757685.954785:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659672960/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757685.954793:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954796:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757685.954798:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954800:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757685.954802:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954804:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757685.954822:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757685.954825:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757685.954829:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757685.954832:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757685.954836:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659672960:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757685.954841:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757685.954843:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757685.954844:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757685.954847:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757685.954849:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757685.954851:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757685.954854:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757685.954858:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757685.954859:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757685.954862:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757685.954864:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757685.954868:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954873:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757685.954875:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787757685.954880:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757685.954891:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757685.954896:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757685.954898:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757685.954900:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954902:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757685.954906:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954911:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305413131 00000100:00000001:2.0:1787757685.954917:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305413825 00000100:00000001:2.0:1787757685.954922:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305413131 00000100:00000001:2.0:1787757685.954926:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305413825 00000100:00000001:2.0:1787757685.954930:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757685.954934:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.954937:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757685.954945:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757685/real 1787757685] req@ffff8ffb4a534a80 x1874588659673088/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757685.954961:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757685.954964:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757685.954969:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757685.954973:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757685.954976:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.954979:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757685.954983:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757685.954991:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659673088/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757685.955006:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.955010:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757685.955016:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.955019:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757685.955022:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.955025:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757685.955045:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757685.955049:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757685.955054:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757685.955057:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757685.955061:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757685.955066:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.955069:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757685.955071:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757685.955073:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757685.955076:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757685.955078:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757685.955080:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757685.955083:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757685.955085:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757685.955111:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757685.955113:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.955116:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757685.955117:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757685.955119:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757685.955122:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757685.955123:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757685.955125:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757685.955127:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757685.955128:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757685.955130:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757685.955132:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.955135:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757685.955137:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757685.955139:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757685.955144:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757685.955146:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757685.955147:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757685.955152:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757685.955155:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757685.955158:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757685.955160:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.955164:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757685.955172:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757685.955174:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757685.955178:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757685.955182:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757686.529294:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757686.529303:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757686.529305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757686.529312:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757686.529317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757686.529321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757686.529325:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757686.529328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757686.529332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757686.721353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757686.721361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757686.721370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757686.721372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757686.721445:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757686.721454:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757686.721461:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757687.553313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757687.553322:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757687.553325:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757687.553331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757687.553336:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757687.553341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757687.553344:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757687.553347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757687.553351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757687.745320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757687.745328:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757687.745335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757687.745338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757687.745423:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757687.745429:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757687.745435:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757688.577380:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757688.577390:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757688.577393:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757688.577400:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757688.577404:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757688.577410:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757688.577413:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757688.577417:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757688.577421:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757688.769311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757688.769320:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757688.769340:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757688.769343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757688.769438:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757688.769443:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757688.769447:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757689.601369:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757689.601381:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757689.601385:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757689.601393:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757689.601398:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757689.601404:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757689.601409:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757689.601412:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757689.601416:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757689.793361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757689.793372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757689.793381:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757689.793383:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757689.793486:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757689.793495:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757689.793500:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757690.625299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757690.625309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757690.625312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757690.625320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757690.625324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757690.625329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757690.625332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757690.625336:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757690.625339:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757690.817338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757690.817346:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757690.817354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757690.817357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757690.817453:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757690.817460:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757690.817465:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757691.073350:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757691.073360:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787757691.073363:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757691.073368:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757691.073374:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757691.073383:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10745 00000100:00080000:0.0:1787757691.073396:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757691.073401:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757691.073404:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757691.073408:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757691.073412:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757691.073415:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787757691.073430:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787757691.073433:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757691.073435:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757691.073441:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757691.073444:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757691.073448:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757691.073452:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757691.073456:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073458:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757691.073461:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757691.073476:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757691.073479:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073483:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757691.073486:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757691.073491:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073506:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757691.073519:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790e680 x1874588659673216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757691.073533:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757691.073539:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073560:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757691.073564:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757691.073569:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787757691.073571:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757691.073573:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757691.073577:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787757691.073580:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10745 00000100:00080000:0.0:1787757691.073584:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757691.073587:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757691.073589:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757691.073592:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757691.073595:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757691.073598:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787757691.073601:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757691.073604:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757691.073607:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757691.073612:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757691.073615:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757691.073617:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757691.073624:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757691.073629:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.073629:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757691.073638:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757691.073640:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:0.0:1787757691.073645:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757691.073647:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757691.073655:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:0.0:1787757691.073656:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.073661:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073663:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.073674:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757691.073676:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757691.073683:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.073684:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757691.073686:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757691.073693:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757691.073694:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757691.073702:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757691.073709:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659673216:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757691.073713:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.073725:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:0.0:1787757691.073733:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790d180 x1874588659673344/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757691.073735:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757691.073742:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.073752:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:0.0:1787757691.073759:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787757691.073763:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073764:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073777:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757691.073781:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757691.073785:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757691.073788:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757691.073790:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757691.073794:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757691.073795:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757691.073803:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00080000:0.0:1787757691.073805:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10745 00000100:00000001:2.0:1787757691.073814:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:0.0:1787757691.073817:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757691.073822:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757691.073824:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787757691.073826:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757691.073828:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757691.073830:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787757691.073832:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757691.073835:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757691.073837:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757691.073841:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757691.073844:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757691.073846:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757691.073850:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757691.073853:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073856:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757691.073859:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757691.073862:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073864:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757691.073871:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757691.073873:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073876:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757691.073878:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757691.073881:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787757691.073888:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757691.073892:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757691.073904:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790fb80 x1874588659673472/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757691.073905:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757691.073914:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757691.073919:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073922:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757691.073924:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757691.073930:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.073937:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.073938:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757691.073949:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757691.073951:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305418945 00000100:00080000:0.0:1787757691.073954:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757691.073961:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787757691.073963:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757691.073967:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757691.073971:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:0.0:1787757691.073972:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10745 00000100:00080000:0.0:1787757691.073985:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757691.073988:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305418945 00000100:00000001:0.0:1787757691.073989:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757691.073996:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757691.074029:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:0.0:1787757691.074046:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757691.074048:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757691.074050:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:0.0:1787757691.074053:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757691.074055:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757691.074057:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757691.074063:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757691.074064:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757691.074067:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757691.074077:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757691.074079:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757691.074082:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757691.074090:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074091:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:0.0:1787757691.074094:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787757691.074100:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787757691.074105:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.074111:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757691.074113:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757691/real 1787757691] req@ffff8ffb4790e680 x1874588659673216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:0.0:1787757691.074118:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757691.074124:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.074130:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757691.074135:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757691.074140:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757691.074153:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:0.0:1787757691.074154:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757691.074161:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:0.0:1787757691.074170:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ed80 x1874588659673600/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757691.074171:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757691.074178:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757691.074183:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757691.074232:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757691.074236:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074247:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757691.074248:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074254:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757691.074267:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659673216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757691.074280:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074287:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757691.074292:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074295:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757691.074297:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074302:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757691.074306:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757691.074311:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757691.074319:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757691.074325:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757691.074332:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659673216:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757691.074342:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074346:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757691.074349:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757691.074354:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757691.074361:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757691.074365:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757691.074368:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757691.074375:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757691.074378:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757691.074383:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757691.074386:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074390:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074401:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757691.074405:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074410:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757691.074412:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757691.074414:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757691.074416:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757691.074418:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.074422:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757691.074425:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757691.074428:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757691.074433:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673344:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757691.074438:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757691.074441:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757691.074443:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.074445:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757691.074450:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074459:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757691.074461:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:2.0:1787757691.074467:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757691.074487:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757691.074494:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757691.074497:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757691.074500:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074503:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757691.074507:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074512:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305418254 00000100:00000001:2.0:1787757691.074516:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305418945 00000100:00000001:2.0:1787757691.074519:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305418254 00000100:00000001:2.0:1787757691.074522:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305418945 00000100:00000001:2.0:1787757691.074525:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757691.074527:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074530:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757691.074535:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757691/real 1787757691] req@ffff8ffb4790d180 x1874588659673344/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757691.074546:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757691.074547:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757691.074550:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757691.074552:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757691.074554:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074557:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757691.074558:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757691.074563:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659673344/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757691.074572:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074575:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757691.074578:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074580:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757691.074582:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074584:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757691.074600:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757691.074603:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757691.074607:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757691.074609:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757691.074614:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673344:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757691.074619:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757691.074621:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757691.074623:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.074626:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757691.074628:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757691.074630:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757691.074633:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673472:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757691.074637:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757691.074639:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757691.074641:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.074644:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757691.074647:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074653:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757691.074655:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:2.0:1787757691.074660:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757691.074672:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757691.074677:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757691.074680:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757691.074682:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074684:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757691.074687:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074691:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305418251 00000100:00000001:2.0:1787757691.074694:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305418945 00000100:00000001:2.0:1787757691.074697:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305418251 00000100:00000001:2.0:1787757691.074700:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305418945 00000100:00000001:2.0:1787757691.074702:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757691.074705:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074707:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757691.074712:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757691/real 1787757691] req@ffff8ffb4790fb80 x1874588659673472/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757691.074729:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757691.074731:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757691.074734:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757691.074735:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757691.074737:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074739:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757691.074741:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757691.074745:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659673472/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757691.074753:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074755:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757691.074758:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074760:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757691.074762:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074764:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757691.074778:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757691.074781:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757691.074785:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757691.074788:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757691.074792:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673472:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757691.074796:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757691.074797:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757691.074799:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.074801:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757691.074803:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757691.074806:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757691.074809:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757691.074812:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757691.074814:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757691.074817:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757691.074819:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757691.074822:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074827:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757691.074829:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:2.0:1787757691.074833:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757691.074844:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757691.074849:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757691.074851:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757691.074853:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074855:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757691.074857:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074861:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305418251 00000100:00000001:2.0:1787757691.074864:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305418945 00000100:00000001:2.0:1787757691.074867:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305418251 00000100:00000001:2.0:1787757691.074870:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305418945 00000100:00000001:2.0:1787757691.074873:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757691.074875:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.074877:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757691.074882:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757691/real 1787757691] req@ffff8ffb4790ed80 x1874588659673600/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757691.074890:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757691.074892:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757691.074894:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757691.074896:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757691.074897:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074899:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757691.074901:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757691.074905:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659673600/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757691.074913:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074916:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757691.074919:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074921:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757691.074923:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074925:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757691.074938:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757691.074942:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757691.074945:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757691.074948:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757691.074952:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757691.074956:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074958:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757691.074959:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757691.074961:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757691.074964:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757691.074966:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757691.074967:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757691.074971:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757691.074972:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757691.074975:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757691.074977:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074979:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757691.074980:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757691.074982:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757691.074984:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757691.074986:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757691.074987:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757691.074990:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757691.074991:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757691.074993:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757691.074994:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.074996:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757691.075028:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757691.075033:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757691.075036:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757691.075039:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757691.075040:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757691.075045:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757691.075048:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757691.075051:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757691.075054:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.075058:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757691.075065:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757691.075069:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757691.075073:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757691.075076:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757691.649288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757691.649297:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757691.649300:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757691.649307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757691.649312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757691.649316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757691.649320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757691.649323:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757691.649327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757691.841343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757691.841355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757691.841364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757691.841368:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757691.841457:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757691.841465:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757691.841472:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757692.673276:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757692.673287:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757692.673292:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757692.673301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757692.673308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757692.673314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757692.673320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757692.673324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757692.673329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757692.865327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757692.865336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757692.865346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757692.865351:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757692.865355:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757692.865360:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757692.865365:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757692.865368:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757692.865371:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757692.865376:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757692.865456:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757692.865463:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757692.865467:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757693.697347:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757693.697365:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757693.697371:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757693.697382:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757693.697390:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757693.697398:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757693.697404:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757693.697410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757693.697416:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757693.889338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757693.889348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757693.889357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757693.889358:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757693.889501:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757693.889510:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757693.889517:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757694.721349:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757694.721361:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757694.721365:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757694.721374:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757694.721380:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757694.721386:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757694.721390:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757694.721394:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757694.721399:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757694.913320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757694.913333:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757694.913338:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757694.913343:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757694.913352:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757694.913355:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757694.913358:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757694.913362:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757694.913369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757694.913372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757694.913516:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757694.913525:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757694.913532:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757695.745373:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757695.745386:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757695.745390:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757695.745398:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757695.745403:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757695.745409:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757695.745413:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757695.745417:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757695.745421:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757695.937355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757695.937367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757695.937378:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757695.937381:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757695.937487:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757695.937500:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757695.937508:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757696.193405:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757696.193418:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757696.193421:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757696.193425:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757696.193431:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757696.193441:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10750 00000100:00080000:2.0:1787757696.193454:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757696.193458:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757696.193461:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757696.193466:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757696.193469:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757696.193473:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757696.193476:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757696.193479:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757696.193481:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757696.193504:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757696.193508:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757696.193512:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757696.193518:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757696.193522:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193525:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757696.193528:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757696.193542:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757696.193544:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193548:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757696.193552:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757696.193557:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193571:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757696.193584:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659673728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757696.193598:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757696.193604:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193631:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757696.193636:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757696.193640:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757696.193642:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757696.193645:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757696.193656:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757696.193659:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10750 00000100:00080000:2.0:1787757696.193663:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757696.193666:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757696.193668:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757696.193672:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757696.193674:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757696.193677:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757696.193681:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757696.193684:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757696.193686:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757696.193691:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757696.193696:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757696.193698:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757696.193703:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757696.193706:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757696.193711:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757696.193713:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.193719:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757696.193720:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757696.193728:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:2.0:1787757696.193732:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757696.193734:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757696.193739:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193744:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.193745:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757696.193750:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757696.193753:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757696.193755:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.193767:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193767:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757696.193769:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757696.193777:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757696.193784:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659673728:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757696.193786:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.193799:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:2.0:1787757696.193802:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659673856/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757696.193807:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757696.193813:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.193824:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757696.193825:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757696.193830:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.193834:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193836:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757696.193846:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757696.193851:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757696.193853:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757696.193856:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757696.193859:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757696.193863:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10750 00000100:00000001:1.0:1787757696.193864:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787757696.193868:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757696.193871:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:2.0:1787757696.193874:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757696.193880:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757696.193881:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787757696.193889:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757696.193892:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757696.193895:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757696.193899:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757696.193902:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757696.193904:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757696.193944:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757696.193949:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757696.193952:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757696.193956:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757696.193958:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193960:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757696.193964:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757696.193973:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:2.0:1787757696.193974:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.193984:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757696.193993:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757696.193996:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787757696.193997:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.194004:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757696.194007:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757696.194008:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787757696.194011:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194016:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.194022:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194023:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757696.194035:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757696.194038:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659673984/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757696.194044:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305424065 00000100:00000001:2.0:1787757696.194060:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757696.194065:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757696.194065:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.194072:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757696.194078:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757696.194080:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305424065 00000100:00080000:2.0:1787757696.194084:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757696.194090:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757696.194091:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757696.194097:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757696.194102:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757696.194105:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10750 00000100:00080000:2.0:1787757696.194110:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757696.194114:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757696.194115:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757696.194123:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757696.194124:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194133:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:2.0:1787757696.194136:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:1.0:1787757696.194147:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757696/real 1787757696] req@ffff8ffb47a39500 x1874588659673728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787757696.194153:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757696.194161:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757696.194174:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787757696.194177:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757696.194234:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757696.194241:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757696.194245:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757696.194247:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757696.194248:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757696.194257:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757696.194258:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757696.194263:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757696.194280:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1787757696.194281:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659673728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757696.194292:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757696.194300:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757696.194307:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757696.194309:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757696.194315:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194320:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757696.194321:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757696.194328:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757696.194331:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:2.0:1787757696.194336:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194342:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757696.194343:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194352:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757696.194355:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757696.194360:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194365:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757696.194365:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757696.194373:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757696.194377:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:2.0:1787757696.194377:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.194388:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757696.194392:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757696.194402:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659674112/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757696.194413:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757696.194423:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757696.194427:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757696.194428:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757696.194435:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757696.194447:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659673728:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757696.194458:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194463:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757696.194468:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757696.194472:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757696.194478:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757696.194481:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757696.194484:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757696.194490:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757696.194493:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757696.194498:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757696.194501:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194505:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194517:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757696.194521:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194525:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757696.194528:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757696.194531:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757696.194534:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757696.194537:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.194542:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757696.194546:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757696.194550:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757696.194558:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673856:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757696.194566:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757696.194570:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757696.194575:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.194578:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757696.194585:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194595:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757696.194596:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:1.0:1787757696.194603:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757696.194624:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757696.194632:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757696.194635:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757696.194637:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194639:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757696.194643:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194649:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305423374 00000100:00000001:1.0:1787757696.194652:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305424065 00000100:00000001:1.0:1787757696.194656:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305423374 00000100:00000001:1.0:1787757696.194659:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305424065 00000100:00000001:1.0:1787757696.194661:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757696.194664:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194666:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757696.194672:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757696/real 1787757696] req@ffff8ffb47a39880 x1874588659673856/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757696.194683:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757696.194685:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757696.194688:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757696.194690:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757696.194692:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194694:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757696.194696:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757696.194701:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659673856/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757696.194710:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194713:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757696.194718:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194720:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757696.194721:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194724:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757696.194739:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757696.194743:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757696.194747:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757696.194750:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757696.194754:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673856:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757696.194759:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757696.194760:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757696.194762:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.194765:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757696.194767:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757696.194769:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757696.194773:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673984:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757696.194778:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757696.194780:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757696.194782:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.194784:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757696.194788:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194794:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757696.194795:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757696.194800:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757696.194813:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757696.194818:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757696.194820:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757696.194822:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194824:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757696.194828:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194832:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305423371 00000100:00000001:1.0:1787757696.194835:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305424065 00000100:00000001:1.0:1787757696.194838:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305423371 00000100:00000001:1.0:1787757696.194841:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305424065 00000100:00000001:1.0:1787757696.194844:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757696.194846:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194849:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757696.194854:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757696/real 1787757696] req@ffff8ffb47a3a680 x1874588659673984/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757696.194863:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757696.194865:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757696.194867:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757696.194869:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757696.194871:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194873:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757696.194874:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757696.194879:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659673984/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757696.194887:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194890:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757696.194893:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194895:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757696.194897:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.194899:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757696.194937:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757696.194941:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757696.194945:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757696.194949:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757696.194953:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659673984:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757696.194958:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757696.194960:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757696.194963:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.194966:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757696.194968:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757696.194971:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757696.194974:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757696.194978:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757696.194980:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757696.194982:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757696.194984:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757696.194988:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.194994:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757696.194996:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757696.195001:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757696.195017:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757696.195022:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757696.195025:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757696.195027:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.195029:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757696.195033:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.195037:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305423371 00000100:00000001:1.0:1787757696.195041:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305424065 00000100:00000001:1.0:1787757696.195043:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305423371 00000100:00000001:1.0:1787757696.195046:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305424065 00000100:00000001:1.0:1787757696.195049:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757696.195051:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.195054:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757696.195059:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757696/real 1787757696] req@ffff8ffb47a38a80 x1874588659674112/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757696.195068:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757696.195070:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757696.195072:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757696.195075:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757696.195077:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195080:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757696.195082:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757696.195086:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659674112/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757696.195095:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195098:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757696.195101:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195103:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757696.195104:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195106:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757696.195119:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757696.195122:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757696.195125:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757696.195127:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757696.195131:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757696.195135:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195138:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757696.195140:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757696.195142:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757696.195145:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757696.195147:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757696.195149:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757696.195152:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757696.195154:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757696.195157:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757696.195158:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195161:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757696.195162:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757696.195164:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757696.195166:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757696.195168:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757696.195169:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757696.195172:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757696.195174:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757696.195177:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757696.195179:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195210:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757696.195213:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757696.195214:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757696.195218:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757696.195220:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757696.195221:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757696.195223:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757696.195225:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757696.195228:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757696.195230:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195233:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757696.195238:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757696.195240:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757696.195242:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757696.195245:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757696.769322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757696.769331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757696.769334:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757696.769341:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757696.769346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757696.769351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757696.769355:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757696.769359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757696.769362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757696.961309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757696.961316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757696.961326:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757696.961329:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757696.961380:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757696.961389:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757696.961396:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757697.793303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757697.793312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757697.793315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757697.793321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757697.793326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757697.793332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757697.793338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757697.793343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757697.793349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757697.985315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757697.985324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757697.985334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757697.985337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757697.985422:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757697.985427:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757697.985433:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757698.817327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757698.817339:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757698.817343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757698.817352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757698.817358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757698.817364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757698.817369:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757698.817373:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757698.817378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757699.009317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757699.009326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757699.009334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757699.009336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757699.009435:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757699.009442:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757699.009447:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757699.841304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757699.841313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757699.841316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757699.841322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757699.841327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757699.841332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757699.841335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757699.841339:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757699.841342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757700.033287:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757700.033295:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757700.033302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757700.033304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757700.033391:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757700.033398:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757700.033402:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757700.865285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757700.865295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757700.865298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757700.865304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757700.865310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757700.865317:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757700.865322:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757700.865327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757700.865333:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757701.057335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757701.057343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757701.057352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757701.057354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757701.057492:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757701.057500:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757701.057507:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757701.313374:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757701.313388:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757701.313393:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757701.313400:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757701.313409:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757701.313422:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10756 00000100:00080000:2.0:1787757701.313440:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757701.313447:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757701.313452:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757701.313459:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757701.313464:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757701.313468:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757701.313473:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757701.313479:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757701.313483:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757701.313494:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757701.313502:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757701.313507:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757701.313514:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757701.313520:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313524:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757701.313529:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757701.313542:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757701.313545:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313549:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757701.313552:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757701.313558:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313574:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757701.313589:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659674240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757701.313604:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757701.313610:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313651:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757701.313662:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757701.313671:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757701.313675:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757701.313680:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757701.313687:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787757701.313689:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787757701.313698:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10756 00000100:00000001:3.0:1787757701.313699:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757701.313705:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757701.313711:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:2.0:1787757701.313713:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757701.313714:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313725:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:3.0:1787757701.313727:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757701.313733:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:3.0:1787757701.313734:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787757701.313745:0:87148:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:3.0:1787757701.313745:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757701.313752:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757701.313758:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757701.313762:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757701.313766:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757701.313769:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757701.313775:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757701.313778:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757701.313781:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757701.313789:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757701.313793:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313796:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757701.313802:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757701.313806:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313809:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757701.313815:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757701.313850:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313853:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757701.313856:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757701.313859:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313868:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757701.313877:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659674368/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757701.313897:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659674240:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757701.313898:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757701.313906:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.313916:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757701.313917:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757701.313924:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:3.0:1787757701.313925:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:2.0:1787757701.313934:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:3.0:1787757701.313935:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313940:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757701.313943:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757701.313947:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757701.313951:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10756 00000100:00080000:2.0:1787757701.313957:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757701.313959:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757701.313961:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757701.313964:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757701.313966:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757701.313969:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757701.313971:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757701.313974:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757701.313975:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757701.313979:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757701.313981:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757701.313983:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757701.313988:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757701.313990:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.313993:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757701.313997:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757701.314000:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314003:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757701.314008:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757701.314010:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314012:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757701.314014:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757701.314016:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314020:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757701.314026:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659674496/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757701.314037:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757701.314040:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314043:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757701.314047:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757701.314050:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757701.314052:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757701.314053:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757701.314056:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757701.314059:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10756 00000100:00080000:2.0:1787757701.314063:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757701.314065:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757701.314066:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757701.314068:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757701.314070:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757701.314072:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757701.314075:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757701.314078:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757701.314080:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757701.314084:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757701.314086:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757701.314089:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757701.314092:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757701.314094:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314097:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757701.314100:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757701.314103:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314105:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757701.314110:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757701.314112:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314114:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757701.314116:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757701.314124:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757701.314125:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757701.314137:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314139:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757701.314152:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659674624/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757701.314161:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757701.314163:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757701.314166:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314167:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757701.314172:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:3.0:1787757701.314180:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757701.314338:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757701.314352:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757701.314359:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757701.314361:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314364:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757701.314371:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314376:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305429186 00000100:00000001:3.0:1787757701.314387:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757701.314394:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305429186 00000100:00000001:3.0:1787757701.314399:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757701.314403:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757701.314406:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314410:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757701.314418:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757701/real 1787757701] req@ffff8ffb47a38380 x1874588659674240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757701.314433:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757701.314435:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757701.314441:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757701.314444:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757701.314446:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314450:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757701.314452:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757701.314458:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659674240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757701.314468:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314472:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757701.314476:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314478:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757701.314479:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314482:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757701.314485:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757701.314489:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757701.314495:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757701.314498:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757701.314503:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659674240:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757701.314510:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314513:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757701.314514:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757701.314518:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757701.314521:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757701.314524:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757701.314526:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757701.314531:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757701.314534:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757701.314539:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757701.314541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314546:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314556:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757701.314559:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314564:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757701.314567:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757701.314570:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757701.314572:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757701.314575:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757701.314580:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757701.314584:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757701.314589:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757701.314595:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674368:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757701.314603:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757701.314607:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757701.314611:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757701.314615:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757701.314621:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314632:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757701.314636:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:3.0:1787757701.314644:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757701.314667:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757701.314674:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757701.314678:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757701.314680:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314683:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757701.314687:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314693:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305428494 00000100:00000001:3.0:1787757701.314697:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305429186 00000100:00000001:3.0:1787757701.314700:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305428494 00000100:00000001:3.0:1787757701.314703:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305429186 00000100:00000001:3.0:1787757701.314706:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757701.314709:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314711:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757701.314717:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757701/real 1787757701] req@ffff8ffb47a38000 x1874588659674368/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757701.314728:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757701.314729:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757701.314732:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757701.314735:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757701.314737:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314739:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757701.314741:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757701.314746:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659674368/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757701.314755:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314757:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757701.314760:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314762:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757701.314764:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314766:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757701.314781:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757701.314784:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757701.314789:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757701.314792:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757701.314797:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674368:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757701.314802:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757701.314804:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757701.314806:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757701.314809:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757701.314811:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757701.314813:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757701.314817:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674496:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757701.314852:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757701.314855:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757701.314857:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757701.314860:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757701.314864:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314873:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757701.314874:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:3.0:1787757701.314880:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757701.314897:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757701.314903:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757701.314906:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757701.314908:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314910:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757701.314914:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314918:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305428491 00000100:00000001:3.0:1787757701.314922:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305429186 00000100:00000001:3.0:1787757701.314925:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305428491 00000100:00000001:3.0:1787757701.314928:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305429186 00000100:00000001:3.0:1787757701.314931:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757701.314933:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.314936:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757701.314941:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757701/real 1787757701] req@ffff8ffb47a3b480 x1874588659674496/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757701.314961:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757701.314963:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757701.314966:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757701.314968:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757701.314970:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314972:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757701.314974:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757701.314979:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659674496/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757701.314988:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314991:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757701.314993:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314995:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757701.314997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.314999:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757701.315011:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757701.315014:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757701.315017:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757701.315020:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757701.315024:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674496:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757701.315028:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757701.315030:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757701.315032:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757701.315035:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757701.315037:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757701.315040:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757701.315043:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757701.315047:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757701.315049:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757701.315051:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757701.315054:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757701.315057:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.315062:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757701.315064:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:3.0:1787757701.315069:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757701.315080:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757701.315085:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757701.315087:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757701.315089:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.315091:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757701.315094:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.315098:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305428491 00000100:00000001:3.0:1787757701.315101:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305429186 00000100:00000001:3.0:1787757701.315104:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305428491 00000100:00000001:3.0:1787757701.315107:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305429186 00000100:00000001:3.0:1787757701.315110:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757701.315112:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.315114:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757701.315118:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757701/real 1787757701] req@ffff8ffb47a39180 x1874588659674624/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757701.315128:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757701.315129:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757701.315132:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757701.315134:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757701.315136:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315138:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757701.315140:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757701.315144:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659674624/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757701.315152:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315154:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757701.315158:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315160:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757701.315162:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315164:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757701.315177:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757701.315180:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757701.315229:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757701.315234:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757701.315241:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757701.315247:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315251:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757701.315253:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757701.315255:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757701.315259:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757701.315262:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757701.315265:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757701.315268:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757701.315270:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757701.315273:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757701.315275:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315277:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757701.315279:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757701.315281:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757701.315283:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757701.315284:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757701.315286:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757701.315289:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757701.315290:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757701.315293:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757701.315294:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315296:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757701.315298:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757701.315299:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757701.315301:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757701.315303:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757701.315304:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757701.315306:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757701.315309:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757701.315312:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757701.315314:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315316:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757701.315321:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757701.315323:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757701.315326:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757701.315328:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757701.889304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757701.889316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757701.889321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757701.889330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757701.889335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757701.889340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757701.889343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757701.889346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757701.889350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757702.081336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757702.081343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757702.081352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757702.081354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757702.081423:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757702.081429:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757702.081434:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757702.913291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757702.913300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757702.913303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757702.913309:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757702.913313:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757702.913318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757702.913321:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757702.913324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757702.913328:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757703.105357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757703.105365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757703.105372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757703.105376:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757703.105380:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757703.105384:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757703.105388:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757703.105391:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757703.105393:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757703.105396:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757703.105532:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757703.105540:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757703.105547:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757703.937287:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757703.937296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757703.937299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757703.937305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757703.937309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757703.937314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757703.937317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757703.937320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757703.937323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757704.129359:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757704.129370:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757704.129378:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757704.129380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757704.129509:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757704.129516:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757704.129521:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757704.961291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757704.961301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757704.961304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757704.961310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757704.961315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757704.961320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757704.961324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757704.961328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757704.961331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757705.153342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757705.153351:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757705.153355:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757705.153359:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757705.153366:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757705.153369:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757705.153372:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757705.153375:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757705.153380:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757705.153382:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757705.153522:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757705.153528:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757705.153533:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757705.985288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757705.985296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757705.985299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757705.985306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757705.985311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757705.985315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757705.985319:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757705.985323:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757705.985326:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757706.177327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757706.177335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757706.177342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757706.177344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757706.177484:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757706.177490:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757706.177495:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757706.433317:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757706.433329:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757706.433332:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757706.433337:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757706.433343:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757706.433352:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10761 00000100:00080000:3.0:1787757706.433365:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757706.433370:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757706.433373:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757706.433378:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757706.433381:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757706.433386:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757706.433389:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757706.433392:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757706.433394:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757706.433400:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757706.433404:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757706.433408:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757706.433412:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757706.433416:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.433418:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757706.433421:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757706.433437:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757706.433439:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.433443:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757706.433447:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757706.433452:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.433466:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757706.433479:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659674752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757706.433493:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757706.433499:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.433528:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757706.433533:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757706.433537:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757706.433539:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757706.433542:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757706.433554:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757706.433557:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10761 00000100:00080000:3.0:1787757706.433561:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757706.433564:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757706.433566:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757706.433569:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757706.433571:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757706.433574:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757706.433577:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757706.433580:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757706.433584:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757706.433589:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757706.433594:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757706.433597:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757706.433605:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757706.433605:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757706.433614:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757706.433614:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.433620:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757706.433622:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757706.433628:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:3.0:1787757706.433630:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757706.433632:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757706.433643:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.433646:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757706.433649:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.433652:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757706.433659:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757706.433668:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659674752:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757706.433670:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757706.433680:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757706.433680:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.433688:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757706.433690:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757706.433694:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757706.433698:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757706.433699:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757706.433710:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.433712:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.433754:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.433762:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757706.433767:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00100000:3.0:1787757706.433777:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659674880/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757706.433778:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757706.433802:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757706.433808:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.433817:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757706.433823:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787757706.433833:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757706.433836:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757706.433842:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757706.433845:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757706.433851:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757706.433853:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757706.433861:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10761 00000100:00000001:1.0:1787757706.433864:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757706.433870:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757706.433872:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757706.433875:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757706.433880:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757706.433886:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757706.433887:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757706.433898:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757706.433901:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305434305 00000020:00000001:3.0:1787757706.433903:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757706.433911:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757706.433915:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757706.433918:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757706.433920:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757706.433921:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757706.433933:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757706.433934:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305434305 00000100:00000001:3.0:1787757706.433945:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.433948:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757706.433952:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757706.433956:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757706.433960:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757706.433963:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757706.433973:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.433976:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757706.433979:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757706.433987:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757706.433992:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.433995:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757706.433998:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757706/real 1787757706] req@ffff8ffb45c30a80 x1874588659674752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757706.434003:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757706.434008:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.434015:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757706.434019:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757706.434026:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.434040:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757706.434043:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757706.434050:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:3.0:1787757706.434054:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659675008/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757706.434061:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757706.434070:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757706.434074:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757706.434078:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757706.434082:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434085:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757706.434089:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434091:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757706.434095:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757706.434104:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787757706.434106:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659674752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757706.434108:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757706.434114:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757706.434125:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757706.434134:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10761 00000100:00000001:1.0:1787757706.434136:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757706.434139:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757706.434146:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757706.434148:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757706.434149:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757706.434157:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757706.434163:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757706.434166:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757706.434167:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757706.434179:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757706.434219:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757706.434221:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757706.434226:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757706.434230:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757706.434233:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757706.434237:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757706.434241:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757706.434244:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757706.434246:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757706.434248:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00000001:3.0:1787757706.434250:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434254:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757706.434257:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434262:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:3.0:1787757706.434267:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757706.434269:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787757706.434275:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757706.434278:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757706.434280:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757706.434286:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757706.434291:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757706.434292:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757706.434299:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757706.434301:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757706.434309:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659674752:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787757706.434318:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659675136/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757706.434323:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434333:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757706.434337:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757706.434341:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757706.434343:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787757706.434349:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.434353:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757706.434356:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757706.434360:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757706.434365:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757706.434372:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757706.434374:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757706.434377:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757706.434378:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434382:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434392:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757706.434395:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434399:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757706.434401:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757706.434403:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757706.434405:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757706.434407:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.434410:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757706.434413:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757706.434416:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757706.434422:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674880:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757706.434426:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757706.434429:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757706.434432:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.434435:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757706.434441:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434454:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757706.434457:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:1.0:1787757706.434465:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757706.434493:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757706.434504:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757706.434509:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757706.434511:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434515:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757706.434521:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434529:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305433614 00000100:00000001:1.0:1787757706.434535:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305434305 00000100:00000001:1.0:1787757706.434541:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305433614 00000100:00000001:1.0:1787757706.434545:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305434305 00000100:00000001:1.0:1787757706.434548:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757706.434551:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434554:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757706.434561:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757706/real 1787757706] req@ffff8ffb45c31c00 x1874588659674880/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757706.434574:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757706.434576:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757706.434581:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757706.434583:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757706.434585:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434588:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757706.434590:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757706.434595:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659674880/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757706.434605:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434608:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757706.434611:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434613:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757706.434615:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434617:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757706.434631:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757706.434635:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757706.434639:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757706.434641:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757706.434646:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659674880:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757706.434651:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757706.434652:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757706.434655:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.434658:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757706.434661:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757706.434663:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757706.434667:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675008:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757706.434671:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757706.434674:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757706.434676:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.434678:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757706.434682:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434689:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757706.434691:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757706.434696:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757706.434713:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757706.434719:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757706.434722:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757706.434724:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434726:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757706.434754:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434759:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305433611 00000100:00000001:1.0:1787757706.434763:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305434305 00000100:00000001:1.0:1787757706.434766:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305433611 00000100:00000001:1.0:1787757706.434769:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305434305 00000100:00000001:1.0:1787757706.434772:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757706.434774:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434777:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757706.434783:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757706/real 1787757706] req@ffff8ffb45c33b80 x1874588659675008/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757706.434793:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757706.434795:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757706.434798:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757706.434800:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757706.434802:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434805:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757706.434807:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757706.434812:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659675008/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757706.434821:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434824:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757706.434826:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434828:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757706.434830:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434832:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757706.434844:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757706.434847:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757706.434851:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757706.434853:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757706.434857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675008:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757706.434861:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757706.434863:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757706.434865:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.434867:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757706.434870:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757706.434873:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757706.434876:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757706.434880:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757706.434882:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757706.434884:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757706.434886:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757706.434890:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434895:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757706.434897:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757706.434902:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757706.434917:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757706.434923:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757706.434925:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757706.434927:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434930:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757706.434932:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434936:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305433611 00000100:00000001:1.0:1787757706.434940:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305434305 00000100:00000001:1.0:1787757706.434942:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305433611 00000100:00000001:1.0:1787757706.434945:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305434305 00000100:00000001:1.0:1787757706.434948:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757706.434951:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.434953:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757706.434958:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757706/real 1787757706] req@ffff8ffb45c30e00 x1874588659675136/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757706.434966:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757706.434968:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757706.434970:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757706.434973:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757706.434975:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.434978:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757706.434980:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757706.434988:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659675136/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757706.435001:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.435005:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757706.435010:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.435013:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757706.435015:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.435019:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757706.435040:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757706.435046:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757706.435054:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757706.435058:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757706.435066:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757706.435075:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.435078:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757706.435080:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757706.435082:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757706.435086:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757706.435088:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757706.435090:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757706.435093:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757706.435095:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757706.435098:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757706.435099:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.435101:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757706.435103:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757706.435105:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757706.435108:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757706.435109:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757706.435111:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757706.435113:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757706.435114:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757706.435116:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757706.435117:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.435119:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757706.435120:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757706.435122:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757706.435125:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757706.435126:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757706.435127:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757706.435130:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757706.435131:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757706.435133:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757706.435136:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.435139:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757706.435146:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757706.435149:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757706.435153:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757706.435156:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757707.009302:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757707.009314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757707.009318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757707.009325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757707.009331:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757707.009337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757707.009340:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757707.009345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757707.009349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757707.201364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757707.201374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757707.201383:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757707.201386:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757707.201444:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757707.201452:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757707.201458:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757708.033278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757708.033287:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757708.033290:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757708.033297:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757708.033301:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757708.033306:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757708.033309:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757708.033313:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757708.033316:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757708.225318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757708.225327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757708.225335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757708.225337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757708.225468:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757708.225474:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757708.225479:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757709.057309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757709.057318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757709.057320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757709.057328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757709.057332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757709.057337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757709.057341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757709.057345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757709.057349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757709.249313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757709.249321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757709.249328:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757709.249331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757709.249415:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757709.249422:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757709.249427:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757710.081391:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757710.081403:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757710.081407:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757710.081415:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757710.081420:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757710.081426:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757710.081430:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757710.081445:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757710.081450:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757710.273323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757710.273334:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757710.273346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757710.273351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757710.273457:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757710.273466:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757710.273473:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757711.105329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757711.105341:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757711.105344:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757711.105352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757711.105358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757711.105364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757711.105367:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757711.105371:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757711.105375:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757711.297301:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757711.297308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757711.297313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757711.297314:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757711.297369:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757711.297377:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757711.297382:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757711.553251:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757711.553260:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757711.553263:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757711.553266:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757711.553269:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757711.553276:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10766 00000100:00080000:3.0:1787757711.553285:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757711.553288:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757711.553290:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757711.553293:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757711.553295:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757711.553298:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757711.553299:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757711.553301:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757711.553302:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757711.553305:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757711.553307:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757711.553309:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757711.553312:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757711.553314:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553327:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757711.553329:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757711.553338:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757711.553339:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553342:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757711.553344:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757711.553348:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553358:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757711.553365:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659675264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757711.553373:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757711.553377:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553400:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757711.553403:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757711.553406:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757711.553407:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757711.553408:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757711.553410:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757711.553411:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10766 00000100:00080000:3.0:1787757711.553415:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757711.553417:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757711.553418:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757711.553420:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757711.553422:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757711.553424:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757711.553425:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757711.553427:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757711.553428:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757711.553430:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757711.553431:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757711.553432:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757711.553436:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757711.553437:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757711.553440:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757711.553441:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757711.553446:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757711.553446:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553450:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553450:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757711.553464:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757711.553465:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:3.0:1787757711.553468:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553470:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757711.553471:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553474:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757711.553476:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757711.553479:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757711.553481:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659675264:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757711.553481:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757711.553484:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757711.553487:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553489:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757711.553492:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553493:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757711.553498:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553502:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787757711.553502:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659675392/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757711.553508:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553511:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757711.553514:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553516:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757711.553526:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757711.553528:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757711.553530:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757711.553531:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757711.553534:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757711.553535:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757711.553539:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00080000:3.0:1787757711.553540:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10766 00000100:00000001:1.0:1787757711.553544:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757711.553545:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757711.553548:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757711.553549:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757711.553551:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757711.553552:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757711.553553:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757711.553554:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757711.553556:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757711.553557:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757711.553559:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757711.553561:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757711.553562:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757711.553564:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757711.553566:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553569:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757711.553571:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757711.553573:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553575:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757711.553580:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757711.553580:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757711.553583:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553587:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757711.553588:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757711.553591:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553592:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757711.553596:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757711.553597:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553599:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553602:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:3.0:1787757711.553607:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659675520/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757711.553608:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553614:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305439425 00000100:00000001:3.0:1787757711.553624:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757711.553625:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757711.553631:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305439425 00000100:00000001:1.0:1787757711.553634:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757711.553636:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553649:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757711.553655:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757711.553657:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553658:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:3.0:1787757711.553663:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757711.553668:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757711/real 1787757711] req@ffff8ffb45c31500 x1874588659675264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757711.553669:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757711.553672:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757711.553675:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757711.553681:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757711.553686:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10766 00000400:00000001:1.0:1787757711.553687:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757711.553691:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:3.0:1787757711.553691:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757711.553696:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757711.553697:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757711.553698:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757711.553702:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000020:00000001:3.0:1787757711.553702:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757711.553706:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757711.553706:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757711.553710:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757711.553711:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:3.0:1787757711.553714:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757711.553715:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757711.553718:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757711.553720:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757711.553722:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659675264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757711.553725:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757711.553731:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757711.553735:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757711.553738:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757711.553739:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757711.553742:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787757711.553743:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553745:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757711.553746:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757711.553748:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00100000:3.0:1787757711.553749:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757711.553750:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787757711.553754:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553755:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757711.553758:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553759:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757711.553764:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:3.0:1787757711.553765:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757711.553768:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757711.553769:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757711.553770:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757711.553773:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1787757711.553773:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757711.553776:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757711.553779:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659675264:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757711.553782:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553784:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553787:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757711.553789:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757711.553792:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00100000:3.0:1787757711.553792:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659675648/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757711.553797:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757711.553800:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757711.553803:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757711.553804:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757711.553807:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553808:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787757711.553809:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553811:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757711.553814:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757711.553815:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553817:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553823:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757711.553824:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553827:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757711.553828:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757711.553829:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757711.553831:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757711.553832:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553834:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757711.553835:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757711.553837:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757711.553840:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675392:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757711.553843:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757711.553844:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757711.553845:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553846:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757711.553848:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553852:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757711.553853:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757711.553857:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757711.553864:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757711.553868:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757711.553869:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757711.553870:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553871:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757711.553873:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553876:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305438733 00000100:00000001:1.0:1787757711.553878:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305439425 00000100:00000001:1.0:1787757711.553880:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305438733 00000100:00000001:1.0:1787757711.553881:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305439425 00000100:00000001:1.0:1787757711.553883:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757711.553884:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553885:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757711.553888:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757711/real 1787757711] req@ffff8ffb45c30700 x1874588659675392/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757711.553893:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757711.553894:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757711.553895:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757711.553896:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757711.553898:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553899:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757711.553900:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757711.553902:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659675392/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757711.553907:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553908:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757711.553909:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553910:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757711.553911:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553912:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757711.553922:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757711.553924:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757711.553926:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757711.553927:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757711.553929:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675392:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757711.553931:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757711.553932:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757711.553933:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553935:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757711.553936:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757711.553937:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757711.553939:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675520:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757711.553941:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757711.553942:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757711.553943:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.553944:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757711.553946:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553949:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757711.553950:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787757711.553952:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757711.553959:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757711.553961:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757711.553963:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757711.553964:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553965:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757711.553966:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553968:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305438730 00000100:00000001:1.0:1787757711.553970:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305439425 00000100:00000001:1.0:1787757711.553972:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305438730 00000100:00000001:1.0:1787757711.553973:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305439425 00000100:00000001:1.0:1787757711.553975:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757711.553976:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.553977:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757711.553980:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757711/real 1787757711] req@ffff8ffb45c32a00 x1874588659675520/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757711.553984:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757711.553985:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757711.553986:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757711.553987:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757711.553988:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553989:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757711.553990:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757711.553992:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659675520/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757711.553996:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.553997:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757711.553999:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554000:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757711.554001:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554002:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757711.554010:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757711.554012:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757711.554014:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757711.554015:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757711.554018:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675520:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757711.554020:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757711.554021:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757711.554022:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.554023:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757711.554024:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757711.554025:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757711.554027:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757711.554029:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757711.554029:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757711.554030:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757711.554031:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757711.554033:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.554044:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757711.554049:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757711.554052:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757711.554058:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757711.554061:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757711.554062:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757711.554063:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.554064:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757711.554066:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.554068:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305438730 00000100:00000001:1.0:1787757711.554070:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305439425 00000100:00000001:1.0:1787757711.554071:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305438730 00000100:00000001:1.0:1787757711.554073:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305439425 00000100:00000001:1.0:1787757711.554074:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757711.554076:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.554077:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757711.554080:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757711/real 1787757711] req@ffff8ffb45c33800 x1874588659675648/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757711.554085:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757711.554086:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757711.554087:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757711.554088:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757711.554089:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554090:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757711.554091:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757711.554094:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659675648/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757711.554097:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554099:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757711.554100:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554101:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757711.554102:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554103:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757711.554112:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757711.554113:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757711.554115:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757711.554117:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757711.554129:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757711.554131:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554132:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757711.554133:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757711.554134:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757711.554135:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757711.554136:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757711.554137:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757711.554138:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757711.554139:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757711.554140:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757711.554141:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554142:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757711.554143:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757711.554144:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757711.554146:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757711.554147:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757711.554148:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757711.554149:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757711.554149:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757711.554150:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757711.554151:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554152:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757711.554153:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757711.554153:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757711.554154:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757711.554155:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757711.554156:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757711.554157:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757711.554157:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757711.554158:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757711.554159:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554160:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757711.554162:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757711.554163:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757711.554164:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757711.554165:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757712.129348:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757712.129359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757712.129363:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757712.129371:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757712.129376:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757712.129383:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757712.129387:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757712.129391:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757712.129395:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757712.321263:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757712.321269:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757712.321274:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757712.321277:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757712.321280:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757712.321283:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757712.321285:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757712.321287:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757712.321288:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757712.321290:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757712.321326:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757712.321330:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757712.321333:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757713.153295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757713.153304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757713.153307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757713.153314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757713.153319:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757713.153324:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757713.153328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757713.153331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757713.153334:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757713.345325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757713.345334:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757713.345341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757713.345343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757713.345421:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757713.345429:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757713.345437:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757714.177310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757714.177320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757714.177323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757714.177329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757714.177334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757714.177338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757714.177341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757714.177344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757714.177348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757714.369314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757714.369324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757714.369332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757714.369336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757714.369461:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757714.369467:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757714.369472:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757715.201320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757715.201329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757715.201332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757715.201339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757715.201343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757715.201348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757715.201352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757715.201356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757715.201359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757715.393342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757715.393353:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757715.393356:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757715.393360:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757715.393366:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757715.393369:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757715.393372:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757715.393375:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757715.393380:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757715.393382:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757715.393463:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757715.393469:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757715.393474:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757716.225296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757716.225305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757716.225309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757716.225315:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757716.225320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757716.225325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757716.225328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757716.225332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757716.225335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757716.417394:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757716.417406:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757716.417416:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757716.417418:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757716.417533:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757716.417539:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757716.417545:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757716.673237:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757716.673246:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757716.673248:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757716.673251:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757716.673255:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757716.673260:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10771 00000100:00080000:3.0:1787757716.673270:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757716.673273:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757716.673274:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757716.673277:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757716.673280:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757716.673282:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757716.673284:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757716.673286:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757716.673287:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757716.673291:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757716.673293:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757716.673295:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757716.673297:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757716.673299:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673301:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757716.673303:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757716.673312:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757716.673313:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673316:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757716.673318:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757716.673321:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673331:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757716.673338:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659675776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757716.673346:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757716.673350:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673362:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757716.673365:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757716.673368:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757716.673369:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757716.673370:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757716.673373:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757716.673374:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10771 00000100:00080000:3.0:1787757716.673377:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757716.673378:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757716.673379:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757716.673381:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757716.673382:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757716.673383:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757716.673385:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757716.673386:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757716.673387:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757716.673389:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757716.673390:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757716.673391:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757716.673394:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757716.673396:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673398:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757716.673403:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757716.673405:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673407:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757716.673409:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757716.673410:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673411:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757716.673412:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757716.673414:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673417:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757716.673421:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659675904/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757716.673427:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757716.673429:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757716.673429:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673436:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757716.673436:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757716.673440:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757716.673441:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757716.673445:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787757716.673445:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757716.673449:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673449:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757716.673452:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757716.673454:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757716.673455:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757716.673458:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10771 02000000:00000001:1.0:1787757716.673459:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757716.673461:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757716.673465:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673465:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757716.673468:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757716.673470:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757716.673473:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:1.0:1787757716.673474:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659675776:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757716.673475:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757716.673478:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757716.673483:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757716.673484:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757716.673485:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757716.673489:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787757716.673489:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787757716.673493:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673494:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757716.673497:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757716.673498:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757716.673502:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757716.673504:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757716.673505:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673506:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757716.673509:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757716.673511:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673512:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757716.673516:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757716.673516:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673517:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757716.673518:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757716.673519:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673522:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757716.673524:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673526:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00100000:3.0:1787757716.673530:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659676032/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757716.673532:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757716.673539:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757716.673540:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673542:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757716.673544:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757716.673546:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757716.673547:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757716.673576:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757716.673578:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757716.673580:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10771 00000100:00080000:3.0:1787757716.673583:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757716.673584:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757716.673585:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:1.0:1787757716.673590:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787757716.673594:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757716.673596:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757716.673598:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757716.673600:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757716.673603:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757716.673603:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757716.673606:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757716.673610:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757716.673611:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757716.673615:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673616:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673620:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757716.673620:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757716.673625:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757716.673627:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757716.673627:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673632:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757716.673634:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757716.673635:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305444545 00000008:00000001:3.0:1787757716.673637:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673641:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757716.673644:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757716.673646:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673647:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757716.673647:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757716.673650:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757716.673652:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673655:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305444545 00000100:00000001:3.0:1787757716.673656:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673661:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00100000:3.0:1787757716.673663:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659676160/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757716.673667:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757716.673672:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673674:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757716.673676:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757716.673677:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757716.673681:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757716.673685:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757716/real 1787757716] req@ffff8ffb45c31180 x1874588659675776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757716.673695:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757716.673697:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757716.673700:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757716.673703:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757716.673705:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673706:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757716.673708:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757716.673711:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659675776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757716.673716:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673719:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757716.673721:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673722:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757716.673723:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673724:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757716.673726:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757716.673728:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757716.673731:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757716.673733:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757716.673736:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659675776:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757716.673739:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673740:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757716.673741:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757716.673743:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757716.673745:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757716.673746:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757716.673747:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757716.673750:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757716.673751:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757716.673753:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757716.673754:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673755:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673760:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757716.673762:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673763:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757716.673764:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757716.673766:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757716.673766:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757716.673768:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757716.673769:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757716.673770:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757716.673772:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757716.673774:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675904:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757716.673776:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757716.673777:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757716.673778:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757716.673779:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757716.673781:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673785:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757716.673786:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:1.0:1787757716.673790:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757716.673797:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757716.673800:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757716.673802:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757716.673803:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673804:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757716.673806:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673808:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305443853 00000100:00000001:1.0:1787757716.673811:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305444545 00000100:00000001:1.0:1787757716.673812:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305443853 00000100:00000001:1.0:1787757716.673814:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305444545 00000100:00000001:1.0:1787757716.673815:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757716.673816:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673818:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757716.673820:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757716/real 1787757716] req@ffff8ffb45c33480 x1874588659675904/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757716.673825:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757716.673826:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757716.673827:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757716.673828:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757716.673829:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673830:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757716.673831:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757716.673834:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659675904/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757716.673838:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673839:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757716.673840:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673841:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757716.673842:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673843:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757716.673853:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757716.673855:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757716.673858:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757716.673859:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757716.673861:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659675904:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757716.673863:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757716.673864:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757716.673865:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757716.673866:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757716.673867:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757716.673868:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757716.673870:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676032:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757716.673872:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757716.673873:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757716.673874:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757716.673875:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757716.673877:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673880:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757716.673881:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787757716.673884:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757716.673890:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757716.673892:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757716.673893:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757716.673894:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673895:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757716.673897:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673899:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305443850 00000100:00000001:1.0:1787757716.673901:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305444545 00000100:00000001:1.0:1787757716.673902:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305443850 00000100:00000001:1.0:1787757716.673904:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305444545 00000100:00000001:1.0:1787757716.673905:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757716.673907:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673908:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757716.673911:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757716/real 1787757716] req@ffff8ffb45c30000 x1874588659676032/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757716.673916:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757716.673917:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757716.673918:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757716.673919:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757716.673920:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673921:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757716.673922:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757716.673925:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659676032/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757716.673929:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673930:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757716.673931:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673932:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757716.673933:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.673934:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757716.673942:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757716.673943:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757716.673945:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757716.673946:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757716.673948:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676032:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757716.673950:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757716.673951:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757716.673952:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757716.673953:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757716.673954:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757716.673956:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757716.673957:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676160:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757716.673961:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757716.673962:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757716.673963:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757716.673964:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757716.673966:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673969:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757716.673970:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757716.673972:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757716.673978:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757716.673980:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757716.673982:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757716.673983:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673984:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757716.673985:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673987:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305443850 00000100:00000001:1.0:1787757716.673989:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305444545 00000100:00000001:1.0:1787757716.673990:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305443850 00000100:00000001:1.0:1787757716.673992:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305444545 00000100:00000001:1.0:1787757716.673993:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757716.673994:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.673996:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757716.673998:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757716/real 1787757716] req@ffff8ffb45c33800 x1874588659676160/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757716.674002:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757716.674003:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757716.674004:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757716.674005:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757716.674006:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674006:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757716.674007:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757716.674009:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659676160/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757716.674018:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674020:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757716.674021:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674022:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757716.674023:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674024:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757716.674032:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757716.674034:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757716.674035:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757716.674037:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757716.674038:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676160:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757716.674041:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674042:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757716.674043:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757716.674044:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757716.674045:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757716.674046:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757716.674047:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757716.674048:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757716.674049:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757716.674050:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757716.674051:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674052:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757716.674053:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757716.674054:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757716.674056:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757716.674056:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757716.674057:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757716.674058:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757716.674059:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757716.674060:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757716.674061:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674061:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757716.674062:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757716.674064:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757716.674065:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757716.674066:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757716.674067:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757716.674068:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757716.674069:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757716.674070:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757716.674070:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674071:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757716.674073:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757716.674074:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757716.674075:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757716.674077:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757717.249262:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757717.249268:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757717.249269:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757717.249273:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757717.249276:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757717.249279:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757717.249281:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757717.249283:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757717.249285:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757717.441440:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757717.441453:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757717.441462:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757717.441465:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757717.441585:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757717.441596:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757717.441604:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757718.273281:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757718.273290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757718.273293:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757718.273299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757718.273304:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757718.273308:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757718.273311:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757718.273315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757718.273318:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757718.465339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757718.465347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757718.465353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757718.465355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757718.465442:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757718.465448:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757718.465453:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757719.297312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757719.297320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757719.297323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757719.297329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757719.297333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757719.297338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757719.297341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757719.297345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757719.297348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757719.489305:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757719.489314:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757719.489323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757719.489326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757719.489422:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757719.489428:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757719.489433:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757720.321317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757720.321328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757720.321332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757720.321343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757720.321349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757720.321356:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757720.321360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757720.321365:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757720.321370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757720.513317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757720.513325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757720.513335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757720.513338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757720.513414:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757720.513420:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757720.513425:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757721.345307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757721.345316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757721.345319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757721.345325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757721.345329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757721.345334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757721.345338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757721.345341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757721.345345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757721.537336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757721.537344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757721.537354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757721.537357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757721.537569:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757721.537578:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757721.537584:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757721.793283:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757721.793296:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757721.793299:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757721.793304:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757721.793310:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757721.793319:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10776 00000100:00080000:3.0:1787757721.793333:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757721.793338:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757721.793340:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757721.793345:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757721.793348:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757721.793352:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757721.793356:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757721.793358:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757721.793361:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757721.793369:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757721.793375:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757721.793380:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757721.793385:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757721.793391:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793396:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757721.793401:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757721.793417:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757721.793420:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793426:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757721.793431:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757721.793438:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793456:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757721.793500:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659676288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757721.793522:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757721.793529:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793557:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757721.793572:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757721.793579:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757721.793582:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757721.793585:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757721.793590:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757721.793596:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10776 00000100:00080000:3.0:1787757721.793606:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757721.793611:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757721.793614:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757721.793618:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:3.0:1787757721.793620:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757721.793626:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:3.0:1787757721.793628:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757721.793632:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:3.0:1787757721.793637:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757721.793639:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757721.793644:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757721.793650:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:1.0:1787757721.793652:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757721.793658:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757721.793661:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787757721.793664:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793668:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787757721.793676:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659676288:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757721.793687:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757721.793689:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757721.793697:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.793698:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757721.793705:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757721.793706:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757721.793712:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757721.793719:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757721.793721:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757721.793728:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793734:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757721.793740:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757721.793745:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.793746:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757721.793748:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.793751:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 02000000:00000001:3.0:1787757721.793759:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757721.793760:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757721.793766:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793771:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757721.793773:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757721.793777:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793787:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757721.793795:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659676416/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757721.793807:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757721.793812:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757721.793817:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757721.793822:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757721.793829:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757721.793835:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757721.793836:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757721.793843:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757721.793846:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757721.793847:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757721.793852:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793854:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757721.793860:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757721.793866:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10776 00000100:00000001:1.0:1787757721.793870:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757721.793875:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757721.793878:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305449665 00000100:00000001:3.0:1787757721.793882:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757721.793887:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757721.793890:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757721.793892:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757721.793897:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787757721.793898:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757721.793909:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757721.793912:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757721.793915:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757721.793917:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305449665 00000100:00080000:3.0:1787757721.793922:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757721.793929:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757721.793933:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757721.793934:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757721.793938:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757721.793943:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.793946:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757721.793946:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757721.793953:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757721.793957:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757721.793959:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793968:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.793970:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787757721.793975:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757721.793980:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.793984:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757721.793987:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787757721.793990:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757721/real 1787757721] req@ffff8ffb45c32a00 x1874588659676288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757721.793990:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.794000:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757721.794014:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659676544/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757721.794028:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757721.794030:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:1.0:1787757721.794037:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757721.794038:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.794045:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757721.794050:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757721.794052:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:3.0:1787757721.794055:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757721.794062:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757721.794062:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757721.794070:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757721.794073:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757721.794075:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757721.794083:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757721.794084:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10776 00000100:00000001:1.0:1787757721.794092:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757721.794094:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757721.794102:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757721.794105:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757721.794110:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757721.794113:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:1.0:1787757721.794115:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659676288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757721.794116:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757721.794124:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757721.794134:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757721.794139:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757721.794149:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757721.794154:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757721.794157:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757721.794160:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794167:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757721.794167:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757721.794175:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757721.794176:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757721.794179:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757721.794182:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794224:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757721.794227:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757721.794233:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757721.794236:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757721.794241:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757721.794244:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00000001:3.0:1787757721.794248:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794251:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757721.794254:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757721.794263:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659676288:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757721.794264:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757721.794270:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794275:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757721.794276:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757721.794283:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757721.794284:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757721.794289:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757721.794291:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.794295:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787757721.794301:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.794304:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757721.794310:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757721.794315:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00100000:3.0:1787757721.794316:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659676672/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757721.794321:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757721.794327:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757721.794333:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757721.794337:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757721.794340:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757721.794342:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757721.794347:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757721.794354:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794359:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757721.794366:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794372:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757721.794377:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757721.794381:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757721.794384:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757721.794388:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.794392:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757721.794413:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757721.794419:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757721.794428:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676416:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757721.794436:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757721.794440:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757721.794443:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.794447:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757721.794453:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794495:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757721.794497:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757721.794504:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757721.794526:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757721.794533:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757721.794537:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757721.794539:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794542:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757721.794546:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794551:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305448974 00000100:00000001:1.0:1787757721.794555:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305449665 00000100:00000001:1.0:1787757721.794558:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305448974 00000100:00000001:1.0:1787757721.794561:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305449665 00000100:00000001:1.0:1787757721.794564:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757721.794567:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794570:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757721.794576:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757721/real 1787757721] req@ffff8ffb45c30700 x1874588659676416/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757721.794588:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757721.794590:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757721.794594:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757721.794596:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757721.794597:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794600:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757721.794602:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757721.794607:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659676416/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757721.794616:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794619:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757721.794622:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794624:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757721.794625:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794627:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757721.794642:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757721.794646:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757721.794650:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757721.794653:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757721.794657:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676416:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757721.794663:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757721.794664:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757721.794666:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.794669:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757721.794671:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757721.794674:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757721.794678:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676544:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757721.794682:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757721.794685:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757721.794687:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.794690:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757721.794693:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794699:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757721.794701:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757721.794706:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757721.794718:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757721.794723:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757721.794726:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757721.794728:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794730:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757721.794733:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794737:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305448971 00000100:00000001:1.0:1787757721.794740:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305449665 00000100:00000001:1.0:1787757721.794743:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305448971 00000100:00000001:1.0:1787757721.794746:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305449665 00000100:00000001:1.0:1787757721.794749:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757721.794751:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794754:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757721.794758:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757721/real 1787757721] req@ffff8ffb45c31500 x1874588659676544/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757721.794767:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757721.794769:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757721.794771:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757721.794773:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757721.794775:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794777:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757721.794778:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757721.794783:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659676544/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757721.794800:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794803:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757721.794806:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794808:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757721.794809:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794811:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757721.794823:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757721.794826:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757721.794830:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757721.794832:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757721.794836:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676544:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757721.794841:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757721.794842:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757721.794844:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.794847:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757721.794849:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757721.794852:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757721.794855:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676672:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757721.794859:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757721.794860:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757721.794862:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757721.794864:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757721.794867:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794872:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757721.794874:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757721.794878:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757721.794890:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757721.794896:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757721.794899:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757721.794902:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794906:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757721.794911:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794917:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305448971 00000100:00000001:1.0:1787757721.794922:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305449665 00000100:00000001:1.0:1787757721.794926:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305448971 00000100:00000001:1.0:1787757721.794930:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305449665 00000100:00000001:1.0:1787757721.794934:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757721.794939:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.794942:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757721.794951:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757721/real 1787757721] req@ffff8ffb45c30e00 x1874588659676672/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757721.794968:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757721.794971:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757721.794977:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757721.794981:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757721.794984:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.794987:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757721.794990:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757721.794998:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659676672/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757721.795009:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.795012:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757721.795015:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.795017:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757721.795019:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.795021:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757721.795040:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757721.795046:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757721.795052:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757721.795057:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757721.795065:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676672:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757721.795072:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.795076:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757721.795079:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757721.795082:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757721.795087:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757721.795091:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757721.795093:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757721.795098:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757721.795100:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757721.795104:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757721.795107:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.795111:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757721.795113:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757721.795115:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757721.795120:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757721.795122:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757721.795123:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757721.795126:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757721.795127:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757721.795130:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757721.795131:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.795133:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757721.795135:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757721.795136:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757721.795139:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757721.795140:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757721.795141:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757721.795143:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757721.795144:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757721.795146:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757721.795148:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.795150:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757721.795155:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757721.795157:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757721.795159:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757721.795161:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757722.369333:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757722.369344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757722.369348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757722.369358:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757722.369364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757722.369370:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757722.369374:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757722.369380:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757722.369386:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757722.561344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757722.561352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757722.561375:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757722.561380:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757722.561385:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757722.561391:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757722.561397:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757722.561401:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757722.561404:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757722.561407:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757722.561536:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757722.561544:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757722.561549:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757723.393332:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757723.393345:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757723.393348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757723.393356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757723.393362:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757723.393368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757723.393372:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757723.393376:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757723.393380:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757723.585364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757723.585376:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757723.585385:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757723.585387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757723.585492:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757723.585500:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757723.585506:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757724.417268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757724.417277:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757724.417279:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757724.417285:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757724.417289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757724.417294:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757724.417297:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757724.417300:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757724.417302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757724.609404:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757724.609436:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757724.609449:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757724.609452:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757724.609593:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757724.609601:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757724.609607:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757725.441467:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757725.441482:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757725.441486:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757725.441495:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757725.441501:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757725.441506:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757725.441509:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757725.441514:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757725.441518:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757725.633335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757725.633347:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757725.633351:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757725.633362:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757725.633369:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757725.633373:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757725.633376:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757725.633379:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757725.633386:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757725.633388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757725.633505:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757725.633514:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757725.633521:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757726.465452:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757726.465469:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757726.465474:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757726.465486:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757726.465493:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757726.465499:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757726.465503:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757726.465508:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757726.465512:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757726.657371:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757726.657408:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757726.657419:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757726.657421:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757726.657577:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757726.657585:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757726.657591:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757726.913284:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757726.913316:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757726.913321:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757726.913327:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757726.913335:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757726.913347:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10781 00000100:00080000:3.0:1787757726.913363:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757726.913416:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757726.913422:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757726.913430:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757726.913433:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757726.913437:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757726.913441:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757726.913444:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757726.913446:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757726.913453:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757726.913456:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757726.913460:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757726.913464:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757726.913468:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913471:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757726.913474:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757726.913486:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757726.913489:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913493:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757726.913497:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757726.913502:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913517:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757726.913529:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659676800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757726.913544:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757726.913549:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913574:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757726.913580:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757726.913587:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757726.913591:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757726.913595:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757726.913600:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757726.913605:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10781 00000100:00080000:3.0:1787757726.913614:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757726.913618:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757726.913621:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757726.913627:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757726.913629:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757726.913633:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757726.913638:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757726.913641:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757726.913642:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757726.913645:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757726.913648:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787757726.913652:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757726.913653:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757726.913657:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913658:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757726.913660:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757726.913662:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757726.913668:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757726.913669:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757726.913674:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757726.913674:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913678:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757726.913679:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757726.913686:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757726.913687:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659676800:192.168.201.118@tcp:250:kworker.0 00000008:00000001:3.0:1787757726.913693:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913696:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757726.913697:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757726.913702:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757726.913705:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757726.913706:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757726.913708:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757726.913711:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757726.913711:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757726.913714:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757726.913717:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757726.913718:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913726:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757726.913737:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659676928/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757726.913741:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757726.913744:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787757726.913751:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757726.913752:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757726.913757:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913762:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757726.913767:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757726.913773:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757726.913776:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757726.913778:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757726.913785:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757726.913790:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10781 00000100:00080000:3.0:1787757726.913796:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757726.913800:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757726.913803:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757726.913807:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757726.913809:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757726.913813:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757726.913816:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757726.913820:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757726.913823:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757726.913829:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757726.913834:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757726.913837:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757726.913841:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757726.913843:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757726.913847:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913849:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305454785 00000100:00000001:3.0:1787757726.913851:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757726.913856:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757726.913861:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913864:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913865:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757726.913871:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757726.913874:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913876:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305454785 00000100:00000001:3.0:1787757726.913878:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757726.913881:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757726.913883:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757726.913884:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913890:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757726.913892:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913895:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913902:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757726.913902:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659677056/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757726.913908:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757726.913917:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757726.913918:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757726.913921:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913922:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:3.0:1787757726.913926:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.913927:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757726.913929:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757726.913932:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:3.0:1787757726.913932:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757726.913935:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757726.913938:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757726.913940:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 02000000:00000001:1.0:1787757726.913942:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757726.913944:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:1.0:1787757726.913947:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757726.913948:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757726.913954:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.913954:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757726.913959:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10781 00000100:00100000:1.0:1787757726.913962:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676928:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1787757726.913966:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757726.913969:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757726.913970:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757726.913974:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787757726.913975:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757726.913979:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757726.913979:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757726.913983:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787757726.913984:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787757726.913987:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:1.0:1787757726.913989:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757726.913992:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757726.913998:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757726.914000:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757726.914002:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757726.914005:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00080000:3.0:1787757726.914007:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757726.914012:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914013:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757726.914016:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757726.914021:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757726.914025:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.914028:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757726.914033:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757726.914040:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914049:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305454093 00000100:00000001:1.0:1787757726.914055:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305454785 00000008:00000001:3.0:1787757726.914056:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914060:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305454093 00000100:00000001:3.0:1787757726.914060:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914065:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305454785 00000100:00000001:1.0:1787757726.914069:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757726.914070:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757726.914073:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757726.914073:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.914077:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757726.914078:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757726.914082:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757726.914083:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757726.914085:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757726.914087:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757726.914092:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757726.914095:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757726.914099:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757726.914100:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757726.914108:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677056:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757726.914116:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757726.914120:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757726.914123:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757726.914124:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659677184/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787757726.914126:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757726.914131:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.914140:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757726.914144:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757726.914151:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914155:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757726.914158:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757726.914163:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757726.914274:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914283:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305454091 00000100:00000001:1.0:1787757726.914289:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305454785 00000100:00000001:1.0:1787757726.914295:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305454091 00000100:00000001:1.0:1787757726.914317:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305454785 00000100:00000001:1.0:1787757726.914323:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757726.914328:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914336:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914339:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757726.914345:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757726.914347:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757726.914350:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757726.914353:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757726.914356:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757726.914359:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757726.914361:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757726.914364:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757726.914387:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757726.914391:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757726.914397:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677184:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757726.914403:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757726.914406:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757726.914408:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757726.914411:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757726.914416:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914427:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757726.914431:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787757726.914436:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757726.914461:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914466:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305454091 00000100:00000001:1.0:1787757726.914470:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305454785 00000100:00000001:1.0:1787757726.914472:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305454091 00000100:00000001:1.0:1787757726.914475:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305454785 00000100:00000001:1.0:1787757726.914478:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757726.914480:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914483:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914486:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757726.914490:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757726.914492:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757726.914495:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757726.914496:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757726.914499:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757726.914501:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757727.489333:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757727.489344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757727.489348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757727.489357:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757727.489416:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757727.489423:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757727.489428:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757727.489433:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757727.489437:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757727.681389:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757727.681398:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757727.681405:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757727.681407:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757727.681488:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757727.681494:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757727.681499:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757728.513396:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757728.513406:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757728.513409:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757728.513415:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757728.513420:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757728.513424:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757728.513428:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757728.513431:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757728.513434:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757728.705314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757728.705325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757728.705334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757728.705545:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757728.705648:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757728.705654:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757728.705660:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757729.537430:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757729.537443:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757729.537446:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757729.537454:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757729.537460:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757729.537466:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.537470:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757729.537475:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.537479:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757729.729345:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757729.729357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757729.729379:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757729.729381:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757729.729476:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757729.729482:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757729.729487:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757729.985860:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1787757729.985947:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757729.985983:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757729.986010:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757729.986046:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757729.986052:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757729.986058:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787757729.986078:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:0.0:1787757729.986092:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757729.986106:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757729.986120:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757729.986121:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757729.986137:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757729.986145:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757729.986155:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:0.0:1787757729.986158:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757726/real 1787757729] req@ffff8ffb45c33b80 x1874588659676800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787757729.986177:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757729.986284:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000400:00000001:0.0:1787757729.986290:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787757729.986297:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000400:00000001:0.0:1787757729.986300:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757729.986307:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757729.986352:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757729.986354:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757729.986366:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757729.986374:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000800:00000100:1.0:1787757729.986384:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:0.0:1787757729.986391:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757729.986401:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757729.986402:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757729.986415:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757729.986420:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757729.986427:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:0.0:1787757729.986432:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659676800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757729.986457:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986465:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757729.986472:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986475:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757729.986478:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986483:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757729.986488:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757729.986518:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757729.986530:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757729.986537:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757729.986545:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659676800:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757729.986556:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757729.986562:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757729.986566:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757729.986572:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757729.986583:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757729.986589:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757729.986594:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757729.986601:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757729.986605:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757729.986611:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757729.986614:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986620:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757729.986625:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757729.986627:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757729.986630:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757729.986643:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757726/real 1787757729] req@ffff8ffb45c30e00 x1874588659677184/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757729.986664:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757729.986666:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757729.986674:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757729.986679:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757729.986682:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986686:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757729.986688:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757729.986698:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659677184/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757729.986713:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986717:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757729.986722:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986726:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757729.986729:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986732:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757729.986763:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757729.986770:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757729.986779:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757729.986784:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757729.986791:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677184:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757729.986801:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757729.986810:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757726/real 1787757729] req@ffff8ffb45c31c00 x1874588659676928/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757729.986827:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757729.986830:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757729.986836:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757729.986840:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757729.986842:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986846:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757729.986849:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757729.986857:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659676928/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757729.986873:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986878:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757729.986883:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986887:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757729.986889:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.986892:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757729.986917:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757729.986923:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757729.986930:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757729.986935:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757729.986941:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659676928:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757729.986950:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757729.986959:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757726/real 1787757729] req@ffff8ffb45c30a80 x1874588659677056/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757729.986977:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757729.986979:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757729.986998:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757729.987002:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757729.987007:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987012:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757729.987015:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757729.987030:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659677056/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757729.987046:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987051:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757729.987058:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987062:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757729.987066:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987069:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757729.987098:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757729.987104:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757729.987113:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757729.987117:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757729.987124:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677056:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757729.987132:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987136:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757729.987138:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757729.987143:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757729.987149:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757729.987153:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757729.987156:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757729.987164:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757729.987166:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757729.987171:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757729.987174:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987178:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757729.987180:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757729.987183:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757729.987246:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757729.987250:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757729.987255:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757729.987264:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757729.987266:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757729.987274:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757729.987281:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987289:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757729.987293:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757729.987297:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757729.987307:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757729.987310:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757729.987332:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757729.987339:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757729.987344:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757729.987349:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757729.987353:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987358:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757729.987375:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757729.987380:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757729.987387:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757729.987390:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757730.561379:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757730.561392:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757730.561395:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757730.561404:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757730.561410:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757730.561416:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757730.561420:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757730.561424:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757730.561428:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757730.753434:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757730.753445:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757730.753454:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757730.753456:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757730.753550:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757730.753559:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757730.753565:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757731.585370:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757731.585380:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757731.585383:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757731.585390:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757731.585394:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757731.585399:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757731.585403:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757731.585406:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757731.585409:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757731.777344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757731.777352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757731.777359:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757731.777361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757731.777436:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757731.777442:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757731.777447:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757732.033362:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757732.033375:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757732.033379:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757732.033384:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757732.033390:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757732.033399:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10786 00000100:00080000:2.0:1787757732.033413:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757732.033417:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757732.033420:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757732.033426:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757732.033429:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757732.033433:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757732.033436:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757732.033439:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757732.033441:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757732.033447:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757732.033450:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757732.033453:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757732.033459:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757732.033462:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.033465:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757732.033468:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757732.033480:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757732.033482:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.033486:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757732.033489:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757732.033495:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.033510:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757732.033521:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659677312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757732.033536:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757732.033541:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.033564:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757732.033568:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757732.033573:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757732.033575:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757732.033577:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757732.033581:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757732.033584:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10786 00000100:00080000:2.0:1787757732.033588:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757732.033591:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757732.033594:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757732.033598:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757732.033600:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757732.033603:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757732.033606:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757732.033608:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757732.033612:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757732.033612:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757732.033623:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:2.0:1787757732.033624:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757732.033631:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757732.033633:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.033638:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787757732.033639:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757732.033644:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757732.033649:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757732.033654:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757732.033657:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.033663:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757732.033663:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757732.033673:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:2.0:1787757732.033674:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757732.033681:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757732.033684:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659677312:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757732.033686:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.033698:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757732.033704:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757732.033708:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.033712:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757732.033718:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757732.033722:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757732.033732:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.033738:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757732.033740:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757732.033745:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.033751:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757732.033757:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.033771:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00100000:2.0:1787757732.033775:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659677440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757732.033785:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757732.033799:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757732.033805:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.033813:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757732.033819:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757732.033826:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757732.033829:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757732.033833:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757732.033840:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000800:00000100:0.0:1787757732.033854:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787757732.033857:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10786 00000100:00080000:2.0:1787757732.033871:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757732.033875:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757732.033878:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757732.033884:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757732.033885:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:2.0:1787757732.033893:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757732.033894:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757732.033898:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757732.033900:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:2.0:1787757732.033904:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757732.033910:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757732.033912:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.033917:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757732.033922:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305459905 00000100:00000001:2.0:1787757732.033923:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757732.033939:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757732.033942:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757732.033946:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757732.033948:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757732.033954:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757732.033962:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305459905 02000000:00000001:2.0:1787757732.033963:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.033973:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757732.033981:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00100000:2.0:1787757732.033982:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757732.033994:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.033996:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757732.033999:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034007:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757732.034015:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757732.034016:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757732.034022:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.034032:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:0.0:1787757732.034036:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757732/real 1787757732] req@ffff8ffb47a3a300 x1874588659677312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757732.034038:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757732.034045:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757732.034060:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757732.034070:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757732.034079:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:2.0:1787757732.034080:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659677568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757732.034091:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757732.034099:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757732.034104:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757732.034107:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757732.034114:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034118:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757732.034124:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034126:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757732.034135:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757732.034147:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659677312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757732.034149:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757732.034157:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757732.034163:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757732.034175:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757732.034181:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.034231:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757732.034237:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.034243:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757732.034247:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.034252:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757732.034256:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757732.034263:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757732.034266:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10786 00000100:00080000:2.0:1787757732.034311:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757732.034315:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757732.034318:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757732.034328:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757732.034329:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757732.034342:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:0.0:1787757732.034348:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659677312:192.168.201.118@tcp:250:kworker.0 00000020:00000001:2.0:1787757732.034350:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757732.034358:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757732.034365:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757732.034367:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757732.034376:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757732.034378:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757732.034383:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:2.0:1787757732.034385:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757732.034393:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00080000:2.0:1787757732.034400:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787757732.034406:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:2.0:1787757732.034409:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757732.034415:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:2.0:1787757732.034417:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000020:00000001:0.0:1787757732.034423:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757732.034428:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757732.034436:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757732.034437:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.034444:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757732.034446:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757732.034453:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757732.034455:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000008:00000001:2.0:1787757732.034460:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034463:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757732.034469:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034474:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757732.034483:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757732.034489:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034491:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757732.034494:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757732.034499:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757732.034502:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757732.034510:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034512:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757732.034520:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757732.034524:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757732.034528:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034529:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757732.034536:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.034544:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:2.0:1787757732.034548:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659677696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757732.034551:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757732.034563:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757732.034580:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677440:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757732.034584:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757732.034595:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034597:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757732.034607:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787757732.034608:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.034612:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.034620:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757732.034629:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034643:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757732.034647:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:0.0:1787757732.034655:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757732.034687:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757732.034699:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757732.034704:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757732.034707:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034712:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757732.034720:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034729:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305459214 00000100:00000001:0.0:1787757732.034736:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305459905 00000100:00000001:0.0:1787757732.034741:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305459214 00000100:00000001:0.0:1787757732.034745:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305459905 00000100:00000001:0.0:1787757732.034750:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757732.034754:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034758:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757732.034768:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757732/real 1787757732] req@ffff8ffb47a38700 x1874588659677440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757732.034798:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757732.034800:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757732.034806:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757732.034810:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757732.034812:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.034816:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757732.034819:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757732.034828:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659677440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757732.034842:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.034847:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757732.034851:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.034855:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757732.034858:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.034861:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757732.034885:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757732.034892:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757732.034898:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757732.034903:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757732.034909:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677440:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757732.034916:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757732.034919:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757732.034922:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.034926:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757732.034929:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757732.034933:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757732.034938:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757732.034944:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757732.034948:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757732.034951:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.034955:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757732.034960:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.034971:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757732.034974:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757732.034981:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757732.035006:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757732.035016:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757732.035021:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757732.035023:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035026:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757732.035032:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035038:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305459211 00000100:00000001:0.0:1787757732.035043:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305459905 00000100:00000001:0.0:1787757732.035048:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305459211 00000100:00000001:0.0:1787757732.035052:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305459905 00000100:00000001:0.0:1787757732.035057:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757732.035060:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035065:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757732.035074:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757732/real 1787757732] req@ffff8ffb47a39180 x1874588659677568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757732.035090:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757732.035093:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757732.035098:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757732.035101:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757732.035104:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035107:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757732.035110:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757732.035117:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659677568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757732.035130:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035134:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757732.035138:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035141:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757732.035144:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035147:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757732.035168:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757732.035173:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757732.035179:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757732.035222:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757732.035231:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757732.035240:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035243:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757732.035246:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757732.035248:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757732.035254:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757732.035257:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757732.035260:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757732.035264:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757732.035267:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757732.035272:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757732.035275:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035300:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757732.035304:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757732.035306:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757732.035310:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757732.035312:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757732.035314:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757732.035318:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757732.035320:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757732.035324:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757732.035327:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035331:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035339:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757732.035342:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035347:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757732.035350:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757732.035353:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757732.035356:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757732.035359:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.035363:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757732.035367:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757732.035371:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757732.035378:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757732.035384:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757732.035388:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757732.035391:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757732.035395:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757732.035400:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035411:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757732.035414:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757732.035422:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757732.035448:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757732.035458:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757732.035462:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757732.035465:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035469:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757732.035474:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035481:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305459212 00000100:00000001:0.0:1787757732.035486:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305459905 00000100:00000001:0.0:1787757732.035490:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305459212 00000100:00000001:0.0:1787757732.035494:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305459905 00000100:00000001:0.0:1787757732.035499:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757732.035503:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035507:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757732.035515:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757732/real 1787757732] req@ffff8ffb47a3b480 x1874588659677696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757732.035532:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757732.035535:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757732.035541:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757732.035544:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757732.035547:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035551:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757732.035554:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757732.035562:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659677696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757732.035576:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035580:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757732.035584:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035587:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757732.035590:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035594:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757732.035615:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757732.035622:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757732.035629:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757732.035634:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757732.035641:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757732.035649:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035653:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757732.035655:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757732.035658:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757732.035663:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757732.035666:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757732.035669:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757732.035674:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757732.035676:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757732.035681:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757732.035683:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035687:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.035693:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757732.035696:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.035699:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757732.035702:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757732.609352:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757732.609366:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757732.609369:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757732.609376:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757732.609381:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757732.609387:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.609391:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757732.609395:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757732.609399:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757732.801414:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757732.801426:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757732.801434:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757732.801441:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757732.801445:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757732.801449:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757732.801454:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757732.801457:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757732.801460:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757732.801463:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757732.801539:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757732.801548:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757732.801554:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757733.633315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757733.633326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757733.633331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757733.633341:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757733.633348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757733.633352:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757733.633356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757733.633359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757733.633363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757733.825313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757733.825321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757733.825329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757733.825332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757733.825429:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757733.825436:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757733.825440:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757734.657338:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757734.657352:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757734.657355:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757734.657364:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757734.657370:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757734.657376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757734.657380:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757734.657384:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757734.657388:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757734.849362:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757734.849373:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757734.849384:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757734.849388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757734.849523:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757734.849531:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757734.849537:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757735.681397:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757735.681407:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757735.681411:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757735.681418:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757735.681425:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757735.681430:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757735.681434:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757735.681438:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757735.681442:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757735.873335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757735.873345:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757735.873349:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757735.873353:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757735.873361:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757735.873366:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757735.873369:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757735.873372:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757735.873377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757735.873379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757735.873479:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757735.873485:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757735.873491:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757736.705322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757736.705331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757736.705334:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757736.705340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757736.705345:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757736.705350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757736.705353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757736.705357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757736.705360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757736.897316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757736.897323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757736.897331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757736.897334:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757736.897443:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757736.897452:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757736.897459:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757737.153323:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757737.153335:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757737.153340:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757737.153345:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757737.153351:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757737.153361:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10791 00000100:00080000:2.0:1787757737.153374:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757737.153380:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757737.153383:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757737.153388:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757737.153391:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757737.153396:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757737.153400:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757737.153404:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757737.153406:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757737.153413:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757737.153417:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757737.153421:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757737.153426:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757737.153431:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.153436:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757737.153441:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757737.153454:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757737.153457:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.153461:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757737.153464:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757737.153470:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.153485:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757737.153498:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659677824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757737.153513:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757737.153520:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.153544:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757737.153548:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757737.153554:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757737.153557:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757737.153560:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757737.153565:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757737.153574:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10791 00000100:00080000:2.0:1787757737.153582:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757737.153585:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757737.153588:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757737.153594:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757737.153597:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757737.153602:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757737.153608:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757737.153627:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757737.153631:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757737.153639:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757737.153648:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.153654:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757737.153654:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757737.153668:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757737.153670:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757737.153676:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757737.153683:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787757737.153683:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.153691:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.153696:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757737.153706:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00100000:2.0:1787757737.153710:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757737.153713:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:2.0:1787757737.153727:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.153730:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.153736:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757737.153750:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:0.0:1787757737.153752:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659677824:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757737.153760:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.153784:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757737.153784:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757737.153802:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757737.153806:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757737.153808:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.153813:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.153820:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757737.153832:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.153835:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757737.153848:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659677952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757737.153862:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757737.153867:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757737.153868:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.153880:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:2.0:1787757737.153887:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.153894:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787757737.153895:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757737.153905:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757737.153908:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757737.153911:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757737.153914:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757737.153918:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10791 00000100:00080000:2.0:1787757737.153924:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757737.153926:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757737.153928:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757737.153931:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757737.153934:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757737.153937:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757737.153940:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757737.153943:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757737.153945:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757737.153950:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757737.153954:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757737.153958:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757737.153963:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757737.153966:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.153968:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757737.153971:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757737.153975:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757737.153987:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757737.153987:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757737.154004:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757737.154006:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.154010:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757737.154013:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757737.154018:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154019:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757737.154027:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154030:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757737.154037:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757737.154040:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659678080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757737.154045:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757737.154062:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757737.154063:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.154069:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154078:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305465025 00000100:00000001:2.0:1787757737.154079:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757737.154094:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757737.154101:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757737.154105:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757737.154107:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:2.0:1787757737.154109:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757737.154117:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757737.154124:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305465025 00000100:00080000:2.0:1787757737.154128:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10791 00000100:00000001:0.0:1787757737.154140:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:2.0:1787757737.154140:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757737.154154:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757737.154156:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757737.154168:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757737.154170:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757737.154247:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757737.154251:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757737.154254:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757737.154261:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757737.154266:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757737.154271:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757737.154280:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757737.154287:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154291:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757737.154293:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757737.154306:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787757737.154316:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757737/real 1787757737] req@ffff8ffb47a38000 x1874588659677824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757737.154318:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.154333:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757737.154343:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757737.154360:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.154384:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757737.154388:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757737.154393:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:2.0:1787757737.154396:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757737.154401:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757737.154406:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757737.154410:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757737.154411:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:2.0:1787757737.154417:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154425:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757737.154433:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757737.154436:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154439:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757737.154446:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757737.154458:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659678208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757737.154471:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659677824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757737.154509:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757737.154521:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154531:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757737.154534:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154543:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757737.154551:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154555:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757737.154558:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154561:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757737.154565:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757737.154571:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757737.154577:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757737.154581:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757737.154587:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659677824:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757737.154594:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154598:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757737.154601:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757737.154605:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757737.154612:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757737.154616:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757737.154619:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757737.154625:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757737.154628:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757737.154633:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757737.154635:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154638:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154649:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757737.154652:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154656:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757737.154658:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757737.154660:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757737.154663:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757737.154665:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.154668:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757737.154672:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757737.154675:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757737.154680:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677952:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757737.154685:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757737.154688:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757737.154691:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.154693:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757737.154697:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154705:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757737.154707:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:0.0:1787757737.154714:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757737.154734:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757737.154741:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757737.154744:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757737.154746:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154749:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757737.154753:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154759:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305464334 00000100:00000001:0.0:1787757737.154763:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305465025 00000100:00000001:0.0:1787757737.154766:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305464334 00000100:00000001:0.0:1787757737.154769:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305465025 00000100:00000001:0.0:1787757737.154772:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757737.154774:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154777:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757737.154782:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757737/real 1787757737] req@ffff8ffb47a38380 x1874588659677952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757737.154792:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757737.154794:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757737.154797:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757737.154799:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757737.154801:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154803:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757737.154804:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757737.154809:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659677952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757737.154820:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154824:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757737.154829:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154833:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757737.154835:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.154838:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757737.154867:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757737.154873:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757737.154881:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757737.154886:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757737.154894:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659677952:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757737.154902:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757737.154906:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757737.154909:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.154914:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757737.154918:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757737.154922:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757737.154929:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757737.154934:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757737.154937:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757737.154939:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.154942:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757737.154946:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154955:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757737.154957:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757737.154963:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757737.154979:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757737.154987:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757737.154989:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757737.154991:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.154994:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757737.154997:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.155002:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305464331 00000100:00000001:0.0:1787757737.155006:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305465025 00000100:00000001:0.0:1787757737.155008:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305464331 00000100:00000001:0.0:1787757737.155012:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305465025 00000100:00000001:0.0:1787757737.155014:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757737.155017:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.155019:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757737.155024:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757737/real 1787757737] req@ffff8ffb47a38a80 x1874588659678080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757737.155043:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757737.155044:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757737.155047:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757737.155049:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757737.155051:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155053:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757737.155055:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757737.155060:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659678080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757737.155069:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155072:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757737.155074:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155076:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757737.155078:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155080:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757737.155099:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757737.155102:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757737.155106:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757737.155108:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757737.155112:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757737.155116:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757737.155118:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757737.155120:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.155123:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757737.155125:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757737.155128:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757737.155132:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757737.155135:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757737.155137:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757737.155140:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757737.155142:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757737.155146:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.155151:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757737.155153:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757737.155158:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757737.155170:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757737.155175:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757737.155177:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757737.155179:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.155181:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757737.155257:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.155266:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305464332 00000100:00000001:0.0:1787757737.155273:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305465025 00000100:00000001:0.0:1787757737.155277:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305464332 00000100:00000001:0.0:1787757737.155282:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305465025 00000100:00000001:0.0:1787757737.155287:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757737.155291:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.155294:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757737.155304:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757737/real 1787757737] req@ffff8ffb47a3a680 x1874588659678208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757737.155318:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757737.155320:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757737.155326:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757737.155329:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757737.155331:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155333:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757737.155336:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757737.155341:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659678208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757737.155351:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155354:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757737.155358:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155360:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757737.155361:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155364:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757737.155383:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757737.155387:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757737.155391:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757737.155393:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757737.155398:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757737.155402:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155404:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757737.155406:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757737.155408:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757737.155411:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757737.155413:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757737.155415:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757737.155418:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757737.155420:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757737.155423:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757737.155424:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155426:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757737.155428:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757737.155430:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757737.155432:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757737.155433:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757737.155435:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757737.155437:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757737.155438:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757737.155440:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757737.155441:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155443:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757737.155444:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757737.155446:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757737.155449:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757737.155450:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757737.155451:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757737.155453:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757737.155455:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757737.155459:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757737.155461:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155463:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.155468:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757737.155470:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.155473:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757737.155475:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757737.729321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757737.729336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757737.729341:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757737.729353:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757737.729361:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757737.729368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.729374:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757737.729380:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757737.729387:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757737.921354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757737.921369:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757737.921387:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757737.921390:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757737.921478:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757737.921487:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757737.921493:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757738.753298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757738.753308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757738.753310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757738.753317:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757738.753322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757738.753326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757738.753330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757738.753333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757738.753337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757738.945302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757738.945313:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757738.945324:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757738.945329:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757738.945410:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757738.945416:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757738.945422:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757739.777312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757739.777322:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757739.777324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757739.777330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757739.777335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757739.777341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757739.777344:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757739.777348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757739.777351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757739.969288:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757739.969297:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757739.969305:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757739.969308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757739.969386:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757739.969392:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757739.969397:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757740.801306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757740.801315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757740.801318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757740.801324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757740.801329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757740.801334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757740.801337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757740.801341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757740.801344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757740.993336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757740.993345:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757740.993352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757740.993354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757740.993463:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757740.993472:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757740.993479:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757741.825321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757741.825332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757741.825335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757741.825343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757741.825361:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757741.825367:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757741.825372:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757741.825376:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757741.825380:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757742.017348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757742.017359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757742.017369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757742.017372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757742.017501:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757742.017510:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757742.017517:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757742.273369:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757742.273384:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757742.273389:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757742.273396:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757742.273405:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757742.273417:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10797 00000100:00080000:3.0:1787757742.273434:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757742.273440:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757742.273443:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757742.273448:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757742.273452:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757742.273456:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757742.273459:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757742.273462:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757742.273464:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757742.273470:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757742.273474:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757742.273478:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757742.273481:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757742.273485:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273488:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757742.273492:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757742.273505:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757742.273507:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273512:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757742.273515:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757742.273520:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273535:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757742.273547:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659678336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757742.273561:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757742.273567:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273588:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757742.273593:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757742.273597:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757742.273600:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757742.273603:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757742.273607:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757742.273610:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10797 00000100:00080000:3.0:1787757742.273616:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757742.273618:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757742.273620:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757742.273623:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757742.273625:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757742.273628:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757742.273631:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757742.273633:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757742.273635:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757742.273639:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757742.273641:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757742.273643:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757742.273649:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757742.273652:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273655:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757742.273662:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757742.273667:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273670:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757742.273675:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757742.273676:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273679:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757742.273681:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757742.273684:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273690:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.273694:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:3.0:1787757742.273699:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659678464/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757742.273702:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757742.273708:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757742.273714:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757742.273720:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757742.273723:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273728:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757742.273731:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757742.273736:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757742.273738:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757742.273746:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787757742.273750:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757742.273751:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757742.273759:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757742.273760:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757742.273769:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10797 02000000:00000001:0.0:1787757742.273771:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757742.273775:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757742.273781:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757742.273784:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757742.273789:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:0.0:1787757742.273792:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659678336:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757742.273793:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757742.273798:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757742.273810:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757742.273818:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787757742.273820:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757742.273823:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273829:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757742.273833:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757742.273837:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757742.273847:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757742.273851:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757742.273860:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757742.273865:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757742.273871:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757742.273877:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757742.273877:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.273885:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787757742.273887:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757742.273896:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757742.273898:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00000001:3.0:1787757742.273908:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273913:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757742.273923:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757742.273925:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273927:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757742.273930:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757742.273933:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.273942:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757742.273950:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659678592/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757742.273961:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757742.273974:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757742.273978:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.273984:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757742.273985:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757742.273993:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757742.273995:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757742.274002:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757742.274003:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757742.274008:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757742.274010:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757742.274015:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757742.274018:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757742.274020:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757742.274029:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305470145 00000100:00080000:3.0:1787757742.274030:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10797 00000100:00080000:3.0:1787757742.274042:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757742.274045:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757742.274048:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757742.274049:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757742.274058:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757742.274061:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305470145 00000020:00000001:3.0:1787757742.274061:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757742.274064:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757742.274071:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757742.274072:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757742.274078:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757742.274082:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274085:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757742.274089:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757742.274093:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787757742.274129:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757742/real 1787757742] req@ffff8ffb45c31500 x1874588659678336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757742.274147:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787757742.274154:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757742.274155:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757742.274160:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757742.274162:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757742.274169:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787757742.274170:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757742.274178:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787757742.274179:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274227:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274231:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757742.274234:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757742.274241:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787757742.274248:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659678336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757742.274251:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757742.274265:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274275:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757742.274277:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274283:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757742.274289:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757742.274292:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757742.274293:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757742.274300:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757742.274302:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274304:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757742.274311:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757742.274313:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787757742.274318:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00080000:0.0:1787757742.274321:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:3.0:1787757742.274324:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757742.274330:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757742.274335:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757742.274337:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757742.274341:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659678336:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757742.274352:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274357:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757742.274361:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00100000:3.0:1787757742.274361:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659678720/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757742.274367:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757742.274373:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757742.274378:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757742.274382:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757742.274388:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787757742.274392:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757742.274393:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757742.274401:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757742.274404:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274405:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274417:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757742.274420:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274428:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757742.274432:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274437:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757742.274440:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757742.274442:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757742.274444:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757742.274446:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757742.274449:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757742.274453:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757742.274456:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757742.274462:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678464:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757742.274468:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757742.274470:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757742.274473:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757742.274475:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757742.274479:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274488:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757742.274490:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757742.274495:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757742.274514:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757742.274521:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757742.274524:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757742.274526:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274528:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757742.274532:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274537:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305469454 00000100:00000001:0.0:1787757742.274541:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305470145 00000100:00000001:0.0:1787757742.274544:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305469454 00000100:00000001:0.0:1787757742.274548:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305470145 00000100:00000001:0.0:1787757742.274550:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757742.274553:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274555:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757742.274561:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757742/real 1787757742] req@ffff8ffb45c30700 x1874588659678464/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757742.274571:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757742.274573:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757742.274576:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757742.274578:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757742.274580:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274582:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757742.274584:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757742.274589:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659678464/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757742.274606:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274609:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757742.274612:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274614:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757742.274616:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274618:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757742.274634:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757742.274637:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757742.274641:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757742.274644:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757742.274647:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678464:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757742.274653:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757742.274654:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757742.274656:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757742.274659:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757742.274661:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757742.274664:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757742.274667:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757742.274671:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757742.274673:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757742.274675:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757742.274677:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757742.274680:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274685:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757742.274687:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757742.274692:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757742.274703:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757742.274708:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757742.274710:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757742.274712:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274715:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757742.274717:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274721:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305469451 00000100:00000001:0.0:1787757742.274725:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305470145 00000100:00000001:0.0:1787757742.274727:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305469451 00000100:00000001:0.0:1787757742.274730:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305470145 00000100:00000001:0.0:1787757742.274733:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757742.274735:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274737:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757742.274742:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757742/real 1787757742] req@ffff8ffb45c32a00 x1874588659678592/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757742.274751:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757742.274752:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757742.274755:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757742.274757:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757742.274758:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274761:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757742.274762:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757742.274769:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659678592/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757742.274783:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274787:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757742.274792:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274795:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757742.274797:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.274800:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757742.274820:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757742.274826:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757742.274833:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757742.274838:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757742.274846:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757742.274854:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757742.274857:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757742.274859:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757742.274864:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757742.274867:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757742.274871:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757742.274877:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757742.274885:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757742.274888:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757742.274890:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757742.274892:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757742.274897:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274905:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757742.274906:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787757742.274912:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757742.274928:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757742.274935:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757742.274938:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757742.274940:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274943:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757742.274946:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274950:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305469451 00000100:00000001:0.0:1787757742.274954:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305470145 00000100:00000001:0.0:1787757742.274957:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305469451 00000100:00000001:0.0:1787757742.274959:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305470145 00000100:00000001:0.0:1787757742.274962:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757742.274965:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.274967:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757742.274972:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757742/real 1787757742] req@ffff8ffb45c33800 x1874588659678720/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757742.274990:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757742.274992:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757742.274995:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757742.274997:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757742.274998:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275000:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757742.275002:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757742.275006:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659678720/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757742.275015:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275018:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757742.275021:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275023:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757742.275024:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275027:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757742.275041:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757742.275044:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757742.275048:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757742.275050:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757742.275054:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757742.275059:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275062:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757742.275064:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757742.275066:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757742.275069:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757742.275071:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757742.275073:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757742.275076:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757742.275077:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757742.275080:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757742.275082:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275084:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757742.275086:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757742.275088:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757742.275090:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757742.275092:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757742.275093:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757742.275095:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757742.275097:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757742.275124:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757742.275127:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275129:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757742.275131:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757742.275132:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757742.275135:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757742.275136:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757742.275138:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757742.275140:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757742.275142:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757742.275144:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757742.275146:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275149:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.275153:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757742.275155:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.275157:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757742.275159:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757742.849290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757742.849298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757742.849302:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757742.849308:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757742.849312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757742.849317:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.849320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757742.849324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757742.849327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757743.041400:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757743.041411:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757743.041422:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757743.041428:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757743.041433:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757743.041438:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757743.041443:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757743.041447:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757743.041450:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757743.041454:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757743.041528:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757743.041534:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757743.041540:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757743.873329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757743.873337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757743.873340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757743.873347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757743.873352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757743.873357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757743.873360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757743.873364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757743.873367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757744.065298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757744.065307:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757744.065315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757744.065318:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757744.065424:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757744.065430:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757744.065435:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757744.897335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757744.897348:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757744.897351:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757744.897360:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757744.897365:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757744.897371:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757744.897375:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757744.897380:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757744.897384:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757745.089344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757745.089357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757745.089368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757745.089371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757745.089476:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757745.089486:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757745.089492:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757745.921355:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757745.921368:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757745.921372:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757745.921380:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757745.921386:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757745.921392:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757745.921395:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757745.921400:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757745.921404:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757746.113373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757746.113387:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757746.113392:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757746.113396:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757746.113404:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757746.113408:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757746.113411:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757746.113415:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757746.113420:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757746.113423:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757746.113509:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757746.113517:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757746.113524:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757746.945380:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757746.945392:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757746.945395:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757746.945405:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757746.945411:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757746.945417:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757746.945421:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757746.945425:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757746.945429:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757747.137364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757747.137376:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757747.137388:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757747.137391:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757747.137476:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757747.137483:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757747.137490:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757747.393324:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757747.393339:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757747.393343:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757747.393349:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757747.393357:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757747.393368:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10802 00000100:00080000:3.0:1787757747.393385:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757747.393391:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757747.393394:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757747.393400:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757747.393404:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757747.393409:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757747.393412:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757747.393415:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757747.393417:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757747.393424:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757747.393427:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757747.393431:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757747.393436:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757747.393441:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393445:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757747.393449:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757747.393467:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757747.393469:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393474:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757747.393479:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757747.393486:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393504:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757747.393519:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659678848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757747.393535:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757747.393541:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393567:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757747.393572:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757747.393577:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757747.393579:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757747.393581:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757747.393585:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757747.393588:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10802 00000100:00080000:3.0:1787757747.393592:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757747.393595:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757747.393597:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757747.393601:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757747.393603:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757747.393606:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757747.393610:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757747.393613:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757747.393616:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757747.393621:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757747.393624:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757747.393626:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757747.393635:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757747.393638:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393641:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757747.393649:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757747.393653:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393657:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.393659:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757747.393662:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757747.393668:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.393669:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757747.393672:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757747.393677:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757747.393678:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757747.393687:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.393688:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757747.393695:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393698:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757747.393704:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757747.393712:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00100000:3.0:1787757747.393713:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659678976/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757747.393721:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757747.393738:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659678848:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757747.393741:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757747.393750:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.393753:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757747.393758:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757747.393763:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787757747.393766:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757747.393769:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757747.393775:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757747.393777:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757747.393783:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757747.393788:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757747.393791:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393793:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757747.393804:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10802 00000100:00080000:3.0:1787757747.393809:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757747.393812:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757747.393814:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757747.393816:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757747.393818:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757747.393820:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757747.393823:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757747.393825:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757747.393827:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757747.393830:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:3.0:1787757747.393833:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757747.393840:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757747.393845:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757747.393852:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757747.393855:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757747.393860:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757747.393863:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393866:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757747.393869:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757747.393873:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393875:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757747.393881:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757747.393882:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393885:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757747.393887:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757747.393890:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393900:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757747.393905:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659679104/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757747.393919:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757747.393929:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757747.393934:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.393938:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757747.393942:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757747.393946:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757747.393948:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757747.393954:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757747.393957:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757747.393958:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757747.393962:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757747.393965:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757747.393973:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10802 00000100:00000001:0.0:1787757747.393974:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757747.393986:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757747.393989:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757747.393994:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305475265 00000100:00000001:3.0:1787757747.393996:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757747.394001:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757747.394046:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787757747.394050:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757747.394056:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757747.394059:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305475265 00000020:00000001:3.0:1787757747.394061:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757747.394069:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757747.394074:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757747.394075:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757747.394082:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757747.394085:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757747.394093:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757747.394096:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.394102:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394105:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787757747.394108:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757747.394116:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757747.394121:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757747.394125:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757747/real 1787757747] req@ffff8ffb45c30000 x1874588659678848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757747.394126:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757747.394132:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757747.394142:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.394149:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757747.394161:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757747.394166:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.394171:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757747.394173:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000400:00000001:0.0:1787757747.394175:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787757747.394177:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.394218:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757747.394226:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659679232/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757747.394237:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757747.394240:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757747.394244:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757747.394255:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757747.394265:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757747.394270:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757747.394274:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394279:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757747.394283:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757747.394297:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659678848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757747.394310:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394315:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757747.394320:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394323:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757747.394324:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394328:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757747.394332:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757747.394336:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757747.394341:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757747.394345:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757747.394351:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659678848:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757747.394357:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394360:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757747.394362:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757747.394366:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757747.394372:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757747.394376:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757747.394379:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757747.394384:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757747.394387:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757747.394390:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757747.394392:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394395:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394407:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757747.394412:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394417:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757747.394420:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757747.394422:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757747.394425:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757747.394428:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757747.394432:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757747.394436:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757747.394439:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757747.394446:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678976:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757747.394454:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757747.394458:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757747.394461:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757747.394465:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757747.394471:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394485:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757747.394488:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:0.0:1787757747.394498:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757747.394526:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757747.394534:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757747.394538:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757747.394540:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394542:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757747.394546:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394553:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305474574 00000100:00000001:0.0:1787757747.394557:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305475265 00000100:00000001:0.0:1787757747.394560:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305474574 00000100:00000001:0.0:1787757747.394563:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305475265 00000100:00000001:0.0:1787757747.394566:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757747.394569:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394571:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757747.394577:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757747/real 1787757747] req@ffff8ffb45c33480 x1874588659678976/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757747.394588:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757747.394590:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757747.394594:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757747.394596:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757747.394598:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394601:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757747.394603:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757747.394607:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659678976/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757747.394616:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394619:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757747.394621:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394623:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757747.394625:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394627:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757747.394649:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757747.394654:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757747.394658:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757747.394661:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757747.394665:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659678976:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757747.394670:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757747.394671:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757747.394673:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757747.394676:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757747.394678:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757747.394680:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757747.394684:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757747.394688:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757747.394691:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757747.394693:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757747.394696:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757747.394700:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394706:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757747.394708:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787757747.394714:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757747.394727:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757747.394732:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757747.394735:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757747.394737:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394740:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757747.394743:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394747:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305474571 00000100:00000001:0.0:1787757747.394750:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305475265 00000100:00000001:0.0:1787757747.394753:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305474571 00000100:00000001:0.0:1787757747.394756:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305475265 00000100:00000001:0.0:1787757747.394759:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757747.394762:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394765:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757747.394770:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757747/real 1787757747] req@ffff8ffb45c31180 x1874588659679104/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757747.394780:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757747.394782:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757747.394784:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757747.394786:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757747.394789:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394791:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757747.394793:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757747.394797:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659679104/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757747.394805:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394808:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757747.394810:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394812:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757747.394814:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394816:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757747.394835:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757747.394838:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757747.394841:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757747.394843:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757747.394847:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757747.394851:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757747.394853:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757747.394855:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757747.394857:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757747.394859:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757747.394861:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757747.394865:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757747.394873:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757747.394875:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757747.394877:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757747.394879:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757747.394883:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394888:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757747.394890:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787757747.394894:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757747.394906:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757747.394910:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757747.394913:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757747.394915:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394917:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757747.394920:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394924:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305474571 00000100:00000001:0.0:1787757747.394927:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305475265 00000100:00000001:0.0:1787757747.394930:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305474571 00000100:00000001:0.0:1787757747.394933:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305475265 00000100:00000001:0.0:1787757747.394936:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757747.394938:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.394941:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757747.394945:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757747/real 1787757747] req@ffff8ffb45c33800 x1874588659679232/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757747.394953:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757747.394955:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757747.394957:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757747.394959:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757747.394961:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394964:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757747.394966:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757747.394970:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659679232/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757747.394987:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394990:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757747.394993:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394995:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757747.394997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.394999:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757747.395043:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757747.395047:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757747.395051:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757747.395053:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757747.395057:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757747.395063:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.395065:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757747.395067:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757747.395070:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757747.395073:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757747.395075:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757747.395078:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757747.395080:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757747.395082:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757747.395085:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757747.395087:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.395089:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757747.395091:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757747.395092:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757747.395095:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757747.395096:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757747.395097:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757747.395099:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757747.395101:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757747.395103:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757747.395104:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.395106:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757747.395107:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757747.395110:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757747.395113:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757747.395116:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757747.395118:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757747.395121:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757747.395124:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757747.395127:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757747.395131:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.395134:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.395141:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757747.395144:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.395147:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757747.395151:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757747.969325:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757747.969335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757747.969338:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757747.969345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757747.969349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757747.969354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.969357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757747.969361:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757747.969364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757748.161306:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757748.161314:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757748.161322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757748.161325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757748.161378:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757748.161385:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757748.161391:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757748.993323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757748.993336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757748.993339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757748.993346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757748.993351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757748.993357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757748.993361:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757748.993366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757748.993370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757749.185300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757749.185310:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757749.185321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757749.185323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757749.185412:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757749.185418:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757749.185423:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757750.017345:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757750.017358:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757750.017362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757750.017370:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757750.017376:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757750.017382:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757750.017385:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757750.017390:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757750.017394:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757750.209301:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757750.209310:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757750.209320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757750.209322:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757750.209408:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757750.209415:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757750.209421:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757751.041300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757751.041312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757751.041316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757751.041323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757751.041329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757751.041335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757751.041339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757751.041343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757751.041347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757751.233334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757751.233344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757751.233354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757751.233357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757751.233481:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757751.233488:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757751.233494:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757752.065300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757752.065313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757752.065316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757752.065325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757752.065331:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757752.065337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.065341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757752.065344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.065348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757752.257322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757752.257331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757752.257341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757752.257349:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757752.257353:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757752.257358:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757752.257362:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757752.257365:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757752.257368:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757752.257371:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757752.257514:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757752.257521:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757752.257526:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757752.513379:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757752.513394:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757752.513397:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757752.513403:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757752.513410:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757752.513421:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10807 00000100:00080000:2.0:1787757752.513439:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757752.513445:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757752.513447:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757752.513454:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757752.513458:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757752.513463:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757752.513467:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757752.513470:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757752.513472:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757752.513478:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757752.513481:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757752.513485:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757752.513490:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757752.513496:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513499:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757752.513503:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757752.513521:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757752.513524:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513529:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757752.513533:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757752.513540:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513560:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757752.513574:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659679360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757752.513590:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757752.513596:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513624:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757752.513629:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757752.513634:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757752.513636:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757752.513638:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757752.513642:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757752.513645:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10807 00000100:00080000:2.0:1787757752.513649:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757752.513652:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757752.513654:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757752.513657:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757752.513658:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757752.513661:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757752.513664:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757752.513666:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757752.513668:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757752.513671:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757752.513674:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757752.513677:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757752.513686:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757752.513689:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513692:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757752.513700:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757752.513705:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513708:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757752.513714:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757752.513716:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513718:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757752.513720:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757752.513723:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757752.513724:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.513736:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757752.513737:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.513745:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:2.0:1787757752.513749:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659679488/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757752.513755:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757752.513763:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513768:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757752.513774:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757752.513777:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757752.513779:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757752.513788:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787757752.513788:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757752.513802:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757752.513803:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513805:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757752.513810:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757752.513816:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:0.0:1787757752.513823:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659679360:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787757752.513824:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10807 00000100:00080000:2.0:1787757752.513830:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757752.513837:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757752.513841:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757752.513843:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:2.0:1787757752.513846:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757752.513850:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757752.513853:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757752.513855:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:2.0:1787757752.513857:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757752.513866:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513866:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757752.513871:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757752.513878:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:2.0:1787757752.513879:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757752.513888:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757752.513890:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757752.513896:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757752.513897:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757752.513909:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513914:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757752.513946:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757752.513953:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513956:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757752.513968:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757752.513970:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.513974:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757752.513977:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757752.513982:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.513984:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757752.513989:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:2.0:1787757752.513995:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.513997:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:2.0:1787757752.514015:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659679616/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757752.514033:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757752.514037:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.514043:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757752.514049:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757752.514054:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757752.514056:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757752.514059:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757752.514063:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757752.514067:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10807 00000800:00000100:0.0:1787757752.514076:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787757752.514081:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757752.514087:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757752.514090:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757752.514093:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757752.514097:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757752.514099:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:2.0:1787757752.514101:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757752.514108:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757752.514111:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757752.514112:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757752.514119:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.514119:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757752.514127:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:2.0:1787757752.514134:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757752.514140:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514142:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757752.514144:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757752.514151:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305480385 02000000:00000001:2.0:1787757752.514152:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757752.514162:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.514165:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757752.514167:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757752.514172:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514174:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757752.514175:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514239:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305480386 02000000:00000001:2.0:1787757752.514245:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757752.514250:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514251:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757752.514254:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757752.514259:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757752.514263:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514272:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757752.514274:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514282:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757752.514291:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659679744/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757752.514292:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757752.514310:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757752/real 1787757752] req@ffff8ffb47a39880 x1874588659679360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757752.514311:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757752.514319:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757752.514328:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757752.514344:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757752.514347:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757752.514353:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757752.514357:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757752.514360:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514364:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757752.514368:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757752.514375:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659679360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757752.514387:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514392:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757752.514396:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514398:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757752.514399:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514403:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757752.514405:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757752.514409:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757752.514414:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757752.514417:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757752.514422:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659679360:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757752.514429:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514433:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757752.514435:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757752.514440:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757752.514447:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757752.514450:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757752.514453:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757752.514459:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757752.514461:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757752.514467:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757752.514469:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514473:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514483:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757752.514486:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514489:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757752.514491:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757752.514493:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757752.514495:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757752.514497:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757752.514500:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757752.514503:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757752.514506:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757752.514510:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679488:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757752.514515:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757752.514517:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757752.514519:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757752.514521:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757752.514525:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514533:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757752.514535:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:0.0:1787757752.514542:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757752.514559:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757752.514565:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757752.514568:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757752.514570:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514572:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757752.514576:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514582:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305479694 00000100:00000001:0.0:1787757752.514587:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305480385 00000100:00000001:0.0:1787757752.514592:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305479694 00000100:00000001:0.0:1787757752.514596:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305480386 00000100:00000001:0.0:1787757752.514600:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757752.514604:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514608:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757752.514616:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757752/real 1787757752] req@ffff8ffb47a39500 x1874588659679488/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757752.514635:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757752.514638:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757752.514644:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757752.514648:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757752.514651:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514654:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757752.514657:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757752.514666:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659679488/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757752.514682:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514686:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757752.514691:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514695:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757752.514698:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514702:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757752.514725:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757752.514729:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757752.514734:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757752.514739:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757752.514747:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679488:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757752.514756:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757752.514759:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757752.514761:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757752.514766:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757752.514770:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757752.514774:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757752.514780:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757752.514787:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757752.514791:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757752.514795:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757752.514799:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757752.514806:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514819:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757752.514822:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757752.514831:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757752.514856:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757752.514866:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757752.514871:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757752.514874:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514877:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757752.514882:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514888:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305479691 00000100:00000001:0.0:1787757752.514893:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305480385 00000100:00000001:0.0:1787757752.514897:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305479691 00000100:00000001:0.0:1787757752.514902:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305480386 00000100:00000001:0.0:1787757752.514906:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757752.514910:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.514914:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757752.514953:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757752/real 1787757752] req@ffff8ffb47a3a680 x1874588659679616/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757752.514973:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757752.514976:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757752.514983:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757752.514986:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757752.514989:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.514995:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757752.514997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757752.515004:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659679616/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757752.515013:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515017:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757752.515021:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515023:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757752.515025:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515028:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757752.515048:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757752.515052:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757752.515056:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757752.515059:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757752.515063:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757752.515068:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757752.515070:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757752.515071:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757752.515074:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757752.515077:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757752.515080:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757752.515084:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757752.515088:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757752.515091:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757752.515093:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757752.515095:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757752.515101:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.515108:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757752.515110:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757752.515116:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757752.515135:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757752.515141:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757752.515144:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757752.515146:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.515148:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757752.515152:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.515157:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305479691 00000100:00000001:0.0:1787757752.515160:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305480385 00000100:00000001:0.0:1787757752.515164:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305479691 00000100:00000001:0.0:1787757752.515167:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305480386 00000100:00000001:0.0:1787757752.515170:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757752.515173:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.515175:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757752.515210:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757752/real 1787757752] req@ffff8ffb47a38a80 x1874588659679744/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757752.515221:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757752.515223:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757752.515227:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757752.515229:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757752.515231:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515234:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757752.515236:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757752.515241:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659679744/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757752.515250:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515253:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757752.515256:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515258:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757752.515260:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515262:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757752.515281:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757752.515284:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757752.515287:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757752.515290:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757752.515294:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659679744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757752.515299:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515301:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757752.515304:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757752.515307:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757752.515310:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757752.515312:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757752.515314:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757752.515319:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757752.515321:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757752.515324:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757752.515326:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515328:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757752.515329:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757752.515331:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757752.515334:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757752.515335:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757752.515336:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757752.515338:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757752.515339:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757752.515343:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757752.515344:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515346:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757752.515348:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757752.515349:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757752.515352:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757752.515353:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757752.515355:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757752.515357:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757752.515358:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757752.515362:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757752.515364:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515366:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757752.515371:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757752.515373:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757752.515376:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757752.515378:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757753.089311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757753.089328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757753.089333:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757753.089357:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757753.089366:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757753.089375:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757753.089382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757753.089388:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757753.089396:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757753.281361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757753.281372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757753.281386:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757753.281389:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757753.281473:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757753.281482:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757753.281488:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757754.113309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757754.113320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757754.113324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757754.113333:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757754.113339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757754.113346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757754.113351:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757754.113355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757754.113361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757754.305361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757754.305370:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757754.305376:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757754.305379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757754.305479:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757754.305488:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757754.305495:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757755.137331:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757755.137340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757755.137342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757755.137349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757755.137354:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757755.137358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757755.137362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757755.137366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757755.137369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757755.329330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757755.329347:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757755.329352:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757755.329360:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757755.329369:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757755.329374:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757755.329379:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757755.329385:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757755.329394:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757755.329397:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757755.329562:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757755.329575:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757755.329584:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757756.161282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757756.161291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757756.161294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757756.161301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757756.161305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757756.161310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757756.161314:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757756.161317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757756.161320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757756.353328:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757756.353336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757756.353345:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757756.353347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757756.353440:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757756.353454:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757756.353480:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757757.185378:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757757.185390:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757757.185394:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757757.185402:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757757.185407:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757757.185414:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.185418:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757757.185422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.185426:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757757.377345:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757757.377354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757757.377361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757757.377363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757757.377444:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757757.377450:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757757.377455:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757757.633409:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757757.633421:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757757.633424:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757757.633429:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757757.633434:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757757.633443:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10812 00000100:00080000:3.0:1787757757.633456:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757757.633462:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757757.633466:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757757.633474:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757757.633478:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757757.633483:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757757.633488:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757757.633491:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757757.633494:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757757.633502:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757757.633508:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757757.633513:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757757.633519:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757757.633526:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.633531:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757757.633535:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757757.633552:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757757.633555:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.633561:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757757.633566:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757757.633572:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.633587:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757757.633601:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659679872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757757.633616:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757757.633622:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.633650:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757757.633657:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757757.633664:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757757.633667:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757757.633670:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757757.633676:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757757.633681:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10812 00000100:00080000:3.0:1787757757.633690:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757757.633694:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757757.633697:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757757.633702:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757757.633705:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757757.633710:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757757.633715:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757757.633719:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757757.633722:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757757.633729:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757757.633735:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757757.633740:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757757.633751:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757757.633757:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.633761:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757757.633766:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787757757.633770:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757757.633775:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757757.633783:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:3.0:1787757757.633783:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.633789:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757757.633793:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.633795:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.633808:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757757.633812:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757757.633814:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757757.633821:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.633822:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757757.633867:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659679872:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757757.633874:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757757.633881:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757757.633883:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757757.633887:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757757.633890:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757757.633894:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.633897:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757757.633909:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.633910:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757757.633929:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659680000/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757757.633938:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757757.633944:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757757.633955:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757757.633964:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757757.633968:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.633975:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757757.633980:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757757.633984:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757757.633987:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757757.633990:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757757.633995:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757757.633998:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10812 00000100:00080000:3.0:1787757757.634006:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757757.634015:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757757.634016:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757757.634025:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757757.634029:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757757.634032:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757757.634036:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757757.634037:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:3.0:1787757757.634041:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757757.634047:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757757.634049:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757757.634050:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757757.634057:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757757.634063:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757757.634064:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757757.634072:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634077:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757757.634078:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757757.634088:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757757.634089:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305485505 02000000:00000001:3.0:1787757757.634100:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.634105:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757757.634109:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00100000:3.0:1787757757.634110:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757757.634120:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.634124:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634126:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305485505 02000000:00000001:3.0:1787757757.634134:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757757.634137:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757757.634141:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634145:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757757.634152:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757757.634153:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757757.634162:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757757.634163:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787757757.634173:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757757.634179:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757757/real 1787757757] req@ffff8ffb45c32a00 x1874588659679872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757757.634246:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757757.634249:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757757.634255:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757757.634257:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634260:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757757.634267:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.634271:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:3.0:1787757757.634275:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659680128/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757757.634277:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757757.634295:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659679872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757757.634300:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757757.634307:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.634317:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634326:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757757.634328:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757757.634337:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757757.634339:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757757.634347:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757757.634348:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757757.634352:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757757.634356:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757757.634359:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757757.634366:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757757.634369:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10812 00000100:00000001:0.0:1787757757.634378:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00080000:3.0:1787757757.634380:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757757.634388:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757757.634389:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757757.634394:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757757.634403:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00080000:0.0:1787757757.634406:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000020:00000001:3.0:1787757757.634407:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757757.634412:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757757.634420:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757757.634424:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00080000:0.0:1787757757.634427:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757757.634428:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757757.634441:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757757.634442:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757757.634446:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757757.634454:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757757.634459:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787757757.634462:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659679872:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757757.634466:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.634477:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757757.634483:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:3.0:1787757757.634485:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757757.634495:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00000001:3.0:1787757757.634498:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634503:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757757.634506:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.634511:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757757.634520:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757757.634524:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:3.0:1787757757.634528:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757757.634532:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:3.0:1787757757.634536:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000020:00000001:0.0:1787757757.634541:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757757.634544:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757757.634552:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757757.634553:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.634562:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757757.634568:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634572:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757757.634579:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757757.634582:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659680256/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757757.634589:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757757.634604:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757757.634608:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757757.634609:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634616:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757757.634619:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634623:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757757.634629:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757757.634632:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757757.634635:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757757.634637:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.634642:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757757.634645:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757757.634648:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757757.634657:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680000:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757757.634663:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757757.634666:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757757.634669:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.634671:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757757.634676:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634686:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757757.634688:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757757.634694:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757757.634714:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757757.634721:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757757.634725:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757757.634727:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634729:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757757.634733:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634738:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305484814 00000100:00000001:0.0:1787757757.634742:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305485505 00000100:00000001:0.0:1787757757.634745:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305484814 00000100:00000001:0.0:1787757757.634748:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305485505 00000100:00000001:0.0:1787757757.634751:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757757.634753:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634756:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757757.634761:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757757/real 1787757757] req@ffff8ffb45c30700 x1874588659680000/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757757.634772:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757757.634773:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757757.634776:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757757.634778:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757757.634780:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.634782:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757757.634784:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757757.634789:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659680000/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757757.634798:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.634801:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757757.634803:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.634805:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757757.634807:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.634810:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757757.634827:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757757.634857:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757757.634861:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757757.634864:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757757.634869:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680000:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757757.634874:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757757.634876:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757757.634878:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.634880:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757757.634883:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757757.634885:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757757.634889:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680128:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757757.634893:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757757.634895:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757757.634897:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.634899:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757757.634903:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634909:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757757.634910:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787757757.634915:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757757.634929:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757757.634935:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757757.634938:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757757.634940:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634942:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757757.634945:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634949:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305484811 00000100:00000001:0.0:1787757757.634953:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305485505 00000100:00000001:0.0:1787757757.634956:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305484811 00000100:00000001:0.0:1787757757.634958:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305485505 00000100:00000001:0.0:1787757757.634963:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757757.634966:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.634968:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757757.634973:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757757/real 1787757757] req@ffff8ffb45c31500 x1874588659680128/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757757.634983:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757757.634985:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757757.634988:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757757.634990:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757757.634991:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.634993:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757757.634995:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757757.635000:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659680128/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757757.635008:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635011:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757757.635014:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635015:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757757.635017:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635019:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757757.635033:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757757.635036:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757757.635039:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757757.635041:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757757.635045:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680128:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757757.635049:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757757.635051:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757757.635053:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.635056:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757757.635058:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757757.635061:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757757.635065:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757757.635068:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757757.635071:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757757.635073:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757757.635075:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757757.635078:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.635083:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757757.635084:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787757757.635089:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757757.635100:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757757.635104:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757757.635107:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757757.635109:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.635111:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757757.635114:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.635118:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305484811 00000100:00000001:0.0:1787757757.635121:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305485505 00000100:00000001:0.0:1787757757.635124:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305484811 00000100:00000001:0.0:1787757757.635127:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305485505 00000100:00000001:0.0:1787757757.635129:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757757.635132:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.635134:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757757.635138:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757757/real 1787757757] req@ffff8ffb45c30a80 x1874588659680256/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757757.635147:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757757.635148:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757757.635151:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757757.635152:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757757.635154:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635156:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757757.635157:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757757.635161:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659680256/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757757.635178:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635181:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757757.635214:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635218:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757757.635219:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635222:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757757.635236:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757757.635240:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757757.635244:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757757.635247:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757757.635251:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757757.635255:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635258:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757757.635259:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757757.635261:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757757.635265:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757757.635267:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757757.635269:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757757.635272:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757757.635274:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757757.635276:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757757.635278:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635280:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757757.635282:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757757.635283:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757757.635285:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757757.635287:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757757.635288:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757757.635290:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757757.635291:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757757.635293:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757757.635295:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635296:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757757.635298:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757757.635300:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757757.635303:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757757.635305:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757757.635306:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757757.635308:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757757.635309:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757757.635311:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757757.635313:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635315:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757757.635319:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757757.635321:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757757.635323:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757757.635325:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757758.209284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757758.209291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757758.209294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757758.209299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757758.209304:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757758.209309:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757758.209312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757758.209315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757758.209319:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757758.401376:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757758.401388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757758.401407:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757758.401412:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757758.401509:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757758.401520:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757758.401530:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757759.233320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757759.233332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757759.233336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757759.233346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757759.233353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757759.233359:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757759.233364:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757759.233368:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757759.233372:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757759.425353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757759.425364:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757759.425373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757759.425375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757759.425463:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757759.425470:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757759.425476:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757760.257370:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757760.257379:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757760.257382:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757760.257389:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757760.257394:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757760.257398:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757760.257402:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757760.257405:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757760.257408:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757760.449341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757760.449348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757760.449357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757760.449359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757760.449456:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757760.449465:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757760.449472:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757761.281300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757761.281309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757761.281312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757761.281319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757761.281324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757761.281328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757761.281332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757761.281335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757761.281339:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757761.473355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757761.473363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757761.473373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757761.473375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757761.473509:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757761.473515:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757761.473520:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757762.305310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757762.305317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757762.305320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757762.305326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757762.305330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757762.305335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.305338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757762.305341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.305345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757762.497319:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757762.497326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757762.497335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757762.497341:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757762.497344:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757762.497349:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757762.497353:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757762.497356:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757762.497359:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757762.497362:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757762.497484:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757762.497490:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757762.497495:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757762.753374:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757762.753389:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757762.753393:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757762.753399:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757762.753406:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757762.753417:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10817 00000100:00080000:1.0:1787757762.753434:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757762.753461:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757762.753466:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757762.753472:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757762.753477:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757762.753483:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757762.753488:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757762.753492:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757762.753495:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757762.753504:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757762.753509:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757762.753515:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757762.753521:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757762.753526:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753529:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757762.753532:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757762.753548:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757762.753550:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753554:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757762.753558:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757762.753563:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753578:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757762.753590:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659680384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757762.753605:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757762.753610:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753631:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757762.753636:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757762.753640:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757762.753642:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757762.753645:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757762.753648:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757762.753651:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10817 00000100:00080000:1.0:1787757762.753658:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757762.753661:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757762.753663:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757762.753666:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757762.753668:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757762.753671:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757762.753674:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757762.753676:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757762.753678:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757762.753681:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757762.753684:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757762.753686:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757762.753692:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757762.753695:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753697:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757762.753703:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757762.753708:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753711:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757762.753716:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757762.753718:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753720:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757762.753723:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757762.753725:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753732:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.753771:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:1.0:1787757762.753776:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659680512/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757762.753782:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757762.753788:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757762.753793:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757762.753797:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757762.753799:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753805:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757762.753810:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757762.753813:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757762.753817:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757762.753821:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757762.753826:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757762.753829:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757762.753835:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00100000:0.0:1787757762.753839:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659680384:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787757762.753840:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757762.753854:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757762.753855:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757762.753862:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10817 02000000:00000001:0.0:1787757762.753863:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757762.753871:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757762.753873:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787757762.753880:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757762.753881:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757762.753886:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757762.753889:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:0.0:1787757762.753891:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757762.753892:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757762.753899:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757762.753905:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757762.753908:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757762.753910:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757762.753915:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757762.753918:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.753919:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787757762.753921:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757762.753926:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 02000000:00000001:1.0:1787757762.753929:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757762.753934:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787757762.753936:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753942:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757762.753945:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757762.753949:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753952:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757762.753960:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757762.753961:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753963:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757762.753965:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757762.753968:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.753976:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757762.753983:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659680640/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757762.753993:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757762.754005:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757762.754009:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.754014:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754017:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787757762.754018:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757762.754027:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757762.754030:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757762.754032:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787757762.754033:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757762.754042:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757762.754044:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757762.754048:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10817 00000100:00000001:0.0:1787757762.754055:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:1.0:1787757762.754058:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757762.754064:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757762.754066:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757762.754070:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757762.754073:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757762.754074:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757762.754078:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757762.754085:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757762.754088:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305490625 00000100:00000001:1.0:1787757762.754089:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757762.754094:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757762.754103:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757762.754105:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757762.754107:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757762.754111:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757762.754113:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:1.0:1787757762.754115:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.754124:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757762.754128:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757762.754132:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305490625 00000008:00000001:1.0:1787757762.754133:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.754138:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754148:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787757762.754149:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757762.754154:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.754158:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757762.754160:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757762.754162:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.754168:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754172:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787757762.754175:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659680768/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757762.754179:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.754247:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757762.754251:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757762.754254:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757762.754262:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757762.754269:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757762/real 1787757762] req@ffff8ffb4a536300 x1874588659680384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757762.754287:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757762.754291:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757762.754300:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757762.754305:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757762.754309:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754312:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757762.754315:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757762.754321:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659680384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757762.754332:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754337:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757762.754341:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754343:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757762.754344:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754348:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757762.754350:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757762.754354:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757762.754359:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757762.754363:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757762.754367:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659680384:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757762.754373:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754376:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757762.754378:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757762.754381:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757762.754387:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757762.754390:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757762.754392:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757762.754398:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757762.754400:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757762.754403:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757762.754406:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754409:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754417:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757762.754419:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754422:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757762.754425:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757762.754427:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757762.754429:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757762.754431:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757762.754433:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757762.754436:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757762.754439:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757762.754443:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680512:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757762.754447:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757762.754450:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757762.754452:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757762.754455:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757762.754459:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754466:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757762.754468:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787757762.754473:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757762.754491:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757762.754498:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757762.754501:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757762.754503:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754505:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757762.754509:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754514:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305489934 00000100:00000001:0.0:1787757762.754518:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305490625 00000100:00000001:0.0:1787757762.754521:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305489934 00000100:00000001:0.0:1787757762.754524:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305490625 00000100:00000001:0.0:1787757762.754527:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757762.754529:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754531:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757762.754536:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757762/real 1787757762] req@ffff8ffb4a537800 x1874588659680512/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757762.754548:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757762.754550:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757762.754553:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757762.754555:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757762.754556:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754558:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757762.754560:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757762.754565:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659680512/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757762.754574:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754576:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757762.754579:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754581:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757762.754582:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754584:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757762.754600:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757762.754603:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757762.754606:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757762.754609:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757762.754613:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680512:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757762.754617:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757762.754618:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757762.754620:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757762.754622:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757762.754624:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757762.754627:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757762.754630:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680640:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757762.754634:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757762.754636:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757762.754639:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757762.754641:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757762.754644:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754649:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757762.754651:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:0.0:1787757762.754656:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757762.754667:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757762.754672:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757762.754674:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757762.754676:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754678:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757762.754681:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754685:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305489931 00000100:00000001:0.0:1787757762.754688:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305490625 00000100:00000001:0.0:1787757762.754691:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305489931 00000100:00000001:0.0:1787757762.754694:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305490625 00000100:00000001:0.0:1787757762.754697:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757762.754699:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754701:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757762.754705:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757762/real 1787757762] req@ffff8ffb4a534a80 x1874588659680640/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757762.754714:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757762.754716:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757762.754718:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757762.754720:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757762.754722:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754724:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757762.754726:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757762.754730:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659680640/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757762.754763:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754767:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757762.754771:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754773:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757762.754775:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754777:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757762.754789:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757762.754792:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757762.754797:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757762.754800:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757762.754804:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680640:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757762.754808:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757762.754810:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757762.754812:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757762.754814:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757762.754816:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757762.754819:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757762.754822:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757762.754827:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757762.754829:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757762.754831:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757762.754833:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757762.754837:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754843:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757762.754845:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787757762.754849:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757762.754864:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757762.754870:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757762.754872:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757762.754875:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754877:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757762.754880:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754884:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305489931 00000100:00000001:0.0:1787757762.754887:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305490625 00000100:00000001:0.0:1787757762.754890:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305489931 00000100:00000001:0.0:1787757762.754893:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305490625 00000100:00000001:0.0:1787757762.754896:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757762.754899:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.754901:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757762.754906:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757762/real 1787757762] req@ffff8ffb4a534700 x1874588659680768/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757762.754915:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757762.754916:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757762.754919:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757762.754921:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757762.754923:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754925:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757762.754927:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757762.754932:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659680768/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757762.754939:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754942:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757762.754945:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754946:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757762.754948:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754950:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757762.754962:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757762.754965:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757762.754969:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757762.754972:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757762.754976:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659680768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757762.754980:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.754982:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757762.754985:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757762.754987:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757762.754990:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757762.754992:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757762.754993:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757762.754997:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757762.754999:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757762.755001:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757762.755003:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.755005:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757762.755007:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757762.755009:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757762.755013:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757762.755015:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757762.755017:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757762.755020:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757762.755022:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757762.755025:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757762.755027:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.755031:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757762.755033:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757762.755035:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757762.755039:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757762.755041:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757762.755043:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757762.755046:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757762.755048:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757762.755051:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757762.755053:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.755057:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757762.755064:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757762.755067:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757762.755071:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757762.755074:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757763.329318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757763.329327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757763.329330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757763.329337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757763.329342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757763.329347:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757763.329351:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757763.329354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757763.329358:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757763.521313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757763.521321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757763.521331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757763.521334:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757763.521390:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757763.521401:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757763.521410:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757764.353296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757764.353305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757764.353308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757764.353314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757764.353318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757764.353323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757764.353326:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757764.353329:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757764.353332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757764.545319:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757764.545327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757764.545334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757764.545336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757764.545456:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757764.545465:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757764.545473:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757765.377345:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757765.377362:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757765.377366:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757765.377378:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757765.377386:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757765.377394:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757765.377400:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757765.377407:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757765.377413:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757765.569379:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757765.569394:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757765.569400:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757765.569407:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757765.569417:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757765.569422:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757765.569426:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757765.569431:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757765.569438:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757765.569440:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757765.569576:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757765.569585:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757765.569591:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757766.401315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757766.401328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757766.401332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757766.401339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757766.401344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757766.401350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757766.401354:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757766.401359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757766.401363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757766.593381:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757766.593390:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757766.593398:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757766.593400:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757766.593541:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757766.593550:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757766.593556:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757767.425310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757767.425323:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757767.425327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757767.425336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757767.425343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757767.425350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.425355:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757767.425360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.425365:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757767.617333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757767.617341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757767.617349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757767.617350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757767.617435:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757767.617443:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757767.617450:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757767.873354:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757767.873366:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757767.873370:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757767.873374:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757767.873380:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757767.873389:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10822 00000100:00080000:3.0:1787757767.873402:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757767.873407:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757767.873409:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757767.873414:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757767.873417:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757767.873421:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757767.873424:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757767.873427:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757767.873430:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757767.873436:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757767.873439:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757767.873443:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757767.873448:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757767.873451:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873454:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757767.873457:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757767.873472:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757767.873474:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873478:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757767.873482:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757767.873487:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873502:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757767.873514:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659680896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757767.873528:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757767.873534:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873557:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757767.873562:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757767.873566:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757767.873568:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757767.873571:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757767.873574:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757767.873577:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10822 00000100:00080000:3.0:1787757767.873581:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757767.873583:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757767.873585:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757767.873588:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757767.873590:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757767.873593:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757767.873595:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757767.873599:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757767.873601:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757767.873605:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757767.873608:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757767.873610:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757767.873616:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757767.873618:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873620:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757767.873627:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757767.873631:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873633:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757767.873638:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757767.873640:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873643:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757767.873645:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757767.873647:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873686:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757767.873696:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659681024/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757767.873708:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757767.873711:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757767.873712:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.873720:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757767.873722:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.873725:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:3.0:1787757767.873731:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757767.873733:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757767.873739:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757767.873744:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757767.873749:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757767.873750:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757767.873757:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757767.873759:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757767.873764:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757767.873767:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757767.873772:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10822 00000100:00080000:3.0:1787757767.873780:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757767.873782:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659680896:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757767.873784:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757767.873790:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757767.873795:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:3.0:1787757767.873798:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757767.873803:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787757767.873805:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787757767.873807:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757767.873811:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757767.873814:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757767.873822:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757767.873824:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757767.873829:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757767.873834:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757767.873836:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757767.873845:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757767.873847:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757767.873852:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757767.873855:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873857:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757767.873862:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00100000:3.0:1787757767.873862:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757767.873870:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000008:00000001:3.0:1787757767.873871:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873882:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.873883:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757767.873892:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757767.873897:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873900:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757767.873902:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757767.873905:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873914:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757767.873922:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659681152/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757767.873932:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757767.873935:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.873938:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757767.873946:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757767.873948:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757767.873958:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757767.873962:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757767.873965:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757767.873968:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757767.873969:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757767.873980:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757767.873981:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10822 00000100:00000001:0.0:1787757767.873987:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757767.873991:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757767.873993:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757767.874001:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757767.874006:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.874007:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757767.874019:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757767.874022:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305495745 00000020:00000001:3.0:1787757767.874023:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757767.874029:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757767.874034:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757767.874038:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757767.874040:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757767.874041:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757767.874053:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757767.874055:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305495745 00000100:00000001:3.0:1787757767.874057:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757767.874066:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757767.874070:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757767.874071:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757767.874081:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757767.874083:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874089:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.874093:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757767.874095:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:3.0:1787757767.874104:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757767.874110:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757767/real 1787757767] req@ffff8ffb45c31c00 x1874588659680896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:3.0:1787757767.874114:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757767.874125:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757767.874142:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:0.0:1787757767.874144:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787757767.874150:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757767.874151:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757767.874159:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757767.874160:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757767.874165:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787757767.874168:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757767.874170:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874177:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757767.874179:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874180:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757767.874232:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659680896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757767.874243:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874248:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757767.874256:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874258:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757767.874261:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757767.874265:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874266:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757767.874272:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757767.874279:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757767.874288:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1787757767.874290:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659681280/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757767.874294:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757767.874307:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659680896:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757767.874324:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874330:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757767.874330:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757767.874337:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757767.874339:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757767.874342:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787757767.874348:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757767.874350:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757767.874357:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757767.874360:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757767.874365:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757767.874368:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757767.874371:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757767.874373:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874378:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874387:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757767.874390:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874393:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757767.874396:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757767.874398:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757767.874400:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757767.874402:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757767.874405:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757767.874407:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757767.874410:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757767.874414:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681024:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757767.874418:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757767.874421:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757767.874423:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757767.874425:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757767.874430:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874437:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757767.874439:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:0.0:1787757767.874444:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757767.874460:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757767.874467:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757767.874470:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757767.874472:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874474:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757767.874478:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874483:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305495054 00000100:00000001:0.0:1787757767.874487:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305495745 00000100:00000001:0.0:1787757767.874490:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305495054 00000100:00000001:0.0:1787757767.874493:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305495745 00000100:00000001:0.0:1787757767.874495:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757767.874498:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874500:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757767.874506:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757767/real 1787757767] req@ffff8ffb45c30e00 x1874588659681024/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757767.874516:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757767.874517:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757767.874520:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757767.874522:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757767.874524:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874526:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757767.874528:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757767.874533:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659681024/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757767.874541:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874544:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757767.874546:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874548:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757767.874550:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874553:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757767.874569:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757767.874572:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757767.874576:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757767.874579:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757767.874583:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681024:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757767.874588:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757767.874589:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757767.874591:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757767.874594:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757767.874596:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757767.874598:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757767.874602:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681152:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757767.874606:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757767.874609:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757767.874610:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757767.874612:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757767.874615:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874621:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757767.874622:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:0.0:1787757767.874627:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757767.874638:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757767.874643:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757767.874645:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757767.874647:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874674:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757767.874678:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874682:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305495051 00000100:00000001:0.0:1787757767.874686:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305495745 00000100:00000001:0.0:1787757767.874689:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305495051 00000100:00000001:0.0:1787757767.874691:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305495745 00000100:00000001:0.0:1787757767.874696:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757767.874699:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874701:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757767.874707:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757767/real 1787757767] req@ffff8ffb45c33b80 x1874588659681152/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757767.874717:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757767.874719:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757767.874722:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757767.874724:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757767.874726:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874728:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757767.874729:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757767.874734:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659681152/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757767.874743:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874745:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757767.874748:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874750:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757767.874751:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874753:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757767.874767:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757767.874770:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757767.874774:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757767.874777:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757767.874781:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681152:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757767.874785:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757767.874787:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757767.874789:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757767.874791:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757767.874793:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757767.874796:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757767.874800:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757767.874803:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757767.874805:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757767.874807:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757767.874810:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757767.874813:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874818:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757767.874820:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787757767.874824:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757767.874837:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757767.874842:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757767.874845:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757767.874847:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874849:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757767.874852:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874856:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305495051 00000100:00000001:0.0:1787757767.874859:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305495745 00000100:00000001:0.0:1787757767.874862:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305495051 00000100:00000001:0.0:1787757767.874865:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305495745 00000100:00000001:0.0:1787757767.874867:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757767.874869:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.874871:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757767.874876:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757767/real 1787757767] req@ffff8ffb45c30a80 x1874588659681280/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757767.874885:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757767.874886:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757767.874889:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757767.874891:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757767.874892:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874894:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757767.874897:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757767.874901:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659681280/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757767.874918:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874921:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757767.874923:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874925:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757767.874927:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874929:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757767.874941:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757767.874944:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757767.874947:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757767.874949:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757767.874953:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757767.874958:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874960:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757767.874962:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757767.874964:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757767.874966:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757767.874968:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757767.874970:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757767.874972:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757767.874974:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757767.874977:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757767.874978:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874980:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757767.874982:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757767.874984:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757767.874986:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757767.874987:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757767.874989:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757767.874991:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757767.874992:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757767.874994:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757767.874995:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.874997:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757767.874999:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757767.875001:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757767.875003:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757767.875005:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757767.875006:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757767.875008:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757767.875010:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757767.875012:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757767.875013:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.875016:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757767.875019:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757767.875021:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757767.875023:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757767.875025:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757768.449300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757768.449310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757768.449312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757768.449319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757768.449324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757768.449328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757768.449331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757768.449334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757768.449338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757768.641350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757768.641359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757768.641369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757768.641372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757768.641468:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757768.641474:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757768.641479:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757769.473287:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757769.473297:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757769.473299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757769.473306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757769.473310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757769.473315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757769.473318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757769.473322:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757769.473326:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757769.665365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757769.665376:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757769.665386:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757769.665389:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757769.665468:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757769.665474:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757769.665479:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757770.497267:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757770.497276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757770.497279:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757770.497286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757770.497291:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757770.497295:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757770.497298:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757770.497301:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757770.497304:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757770.689337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757770.689345:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757770.689352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757770.689354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757770.689449:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757770.689456:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757770.689461:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757771.521263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757771.521271:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757771.521273:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757771.521279:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757771.521283:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757771.521287:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757771.521290:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757771.521294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757771.521297:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757771.713261:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757771.713267:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757771.713271:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757771.713272:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757771.713308:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757771.713312:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757771.713314:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757772.545310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757772.545321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757772.545323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757772.545365:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757772.545372:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757772.545378:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.545382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757772.545387:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.545391:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757772.737358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757772.737369:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757772.737395:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757772.737404:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757772.737409:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757772.737414:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757772.737419:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757772.737422:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757772.737425:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757772.737428:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757772.737526:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757772.737535:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757772.737550:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757772.993394:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757772.993412:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757772.993418:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757772.993425:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757772.993435:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757772.993447:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10827 00000100:00080000:2.0:1787757772.993467:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757772.993475:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757772.993482:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757772.993490:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757772.993495:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757772.993501:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757772.993506:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757772.993511:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757772.993513:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757772.993522:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757772.993526:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757772.993531:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757772.993536:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757772.993541:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993545:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757772.993549:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757772.993583:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757772.993586:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993591:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757772.993595:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757772.993601:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993616:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757772.993629:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659681408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757772.993644:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757772.993650:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993669:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757772.993675:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757772.993680:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757772.993683:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757772.993691:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757772.993693:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757772.993699:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757772.993706:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:2.0:1787757772.993713:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10827 00000100:00000001:0.0:1787757772.993717:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757772.993727:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:2.0:1787757772.993729:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757772.993733:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993748:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757772.993752:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757772.993757:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757772.993760:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757772.993775:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757772.993777:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757772.993785:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:0.0:1787757772.993795:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659681408:192.168.201.118@tcp:250:kworker.0 00000020:00000001:2.0:1787757772.993797:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757772.993816:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:2.0:1787757772.993817:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:0.0:1787757772.993828:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757772.993833:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.993838:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757772.993845:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993849:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757772.993857:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757772.993875:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757772.993878:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757772.993886:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:2.0:1787757772.993889:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.993895:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787757772.993899:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757772.993914:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757772.993922:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993925:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757772.993933:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757772.993941:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993946:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757772.993955:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757772.993958:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993962:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757772.993964:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757772.993969:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.993978:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757772.993990:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659681536/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757772.994007:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757772.994015:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757772.994021:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.994029:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757772.994034:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757772.994038:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:2.0:1787757772.994041:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757772.994046:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757772.994051:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757772.994054:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757772.994058:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757772.994064:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757772.994067:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10827 00000100:00000001:0.0:1787757772.994082:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:2.0:1787757772.994084:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757772.994092:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757772.994094:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757772.994098:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757772.994101:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757772.994103:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757772.994110:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757772.994114:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305500865 00000020:00000001:2.0:1787757772.994120:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757772.994126:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757772.994129:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757772.994133:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:2.0:1787757772.994135:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757772.994143:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994149:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305500865 02000000:00000001:2.0:1787757772.994151:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757772.994163:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757772.994168:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757772.994171:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.994175:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757772.994178:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00100000:2.0:1787757772.994256:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757772.994263:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.994266:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757772.994273:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757772.994275:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757772.994277:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.994290:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757772.994294:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757772.994296:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757772.994305:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757772.994317:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757772/real 1787757772] req@ffff8ffb47a38380 x1874588659681408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757772.994321:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757772.994347:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659681664/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787757772.994372:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757772.994375:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:0.0:1787757772.994378:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757772.994388:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994393:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757772.994401:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994402:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757772.994408:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.994416:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:2.0:1787757772.994417:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757772.994420:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757772.994440:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659681408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757772.994441:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757772.994454:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757772.994463:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757772.994465:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757772.994473:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757772.994478:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757772.994480:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.994488:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757772.994491:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.994500:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00080000:2.0:1787757772.994500:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10827 00000100:00080000:2.0:1787757772.994515:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757772.994520:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757772.994522:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757772.994526:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757772.994532:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:2.0:1787757772.994533:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757772.994540:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757772.994550:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:0.0:1787757772.994554:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:2.0:1787757772.994555:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757772.994583:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787757772.994594:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1787757772.994601:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757772.994613:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994614:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:2.0:1787757772.994618:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757772.994630:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757772.994638:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757772.994643:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659681408:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757772.994644:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757772.994652:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757772.994663:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994664:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757772.994672:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994674:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757772.994685:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757772.994688:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757772.994693:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.994694:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:2.0:1787757772.994710:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757772.994714:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:2.0:1787757772.994714:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757772.994737:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757772.994739:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757772.994753:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:2.0:1787757772.994760:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.994763:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757772.994771:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757772.994778:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00100000:2.0:1787757772.994780:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659681792/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757772.994786:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.994802:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.994824:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757772.994840:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757772.994840:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757772.994858:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994860:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757772.994869:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757772.994874:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757772.994874:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994882:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757772.994887:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.994893:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757772.994898:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757772.994903:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757772.994911:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681536:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757772.994917:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757772.994923:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757772.994926:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.994928:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757772.994934:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994944:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757772.994946:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:0.0:1787757772.994952:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757772.994975:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757772.994984:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757772.994989:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757772.994992:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.994995:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757772.995001:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995009:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305500174 00000100:00000001:0.0:1787757772.995015:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305500865 00000100:00000001:0.0:1787757772.995020:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305500174 00000100:00000001:0.0:1787757772.995025:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305500865 00000100:00000001:0.0:1787757772.995029:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757772.995033:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995037:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757772.995047:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757772/real 1787757772] req@ffff8ffb47a38000 x1874588659681536/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757772.995069:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757772.995072:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757772.995078:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757772.995081:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757772.995085:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995088:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757772.995090:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757772.995097:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659681536/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757772.995107:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995110:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757772.995113:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995115:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757772.995117:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995119:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757772.995141:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757772.995144:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757772.995149:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757772.995151:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757772.995155:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681536:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757772.995161:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757772.995164:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757772.995167:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.995170:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757772.995172:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757772.995175:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757772.995179:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681664:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757772.995247:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757772.995254:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757772.995259:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.995267:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757772.995274:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995287:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757772.995290:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757772.995299:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757772.995320:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757772.995328:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757772.995331:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757772.995333:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995335:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757772.995339:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995344:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305500172 00000100:00000001:0.0:1787757772.995348:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305500865 00000100:00000001:0.0:1787757772.995351:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305500172 00000100:00000001:0.0:1787757772.995354:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305500865 00000100:00000001:0.0:1787757772.995357:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757772.995359:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995362:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757772.995367:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757772/real 1787757772] req@ffff8ffb47a3b480 x1874588659681664/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757772.995377:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757772.995378:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757772.995381:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757772.995383:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757772.995385:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995388:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757772.995390:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757772.995395:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659681664/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757772.995403:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995406:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757772.995410:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995413:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757772.995414:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995417:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757772.995436:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757772.995440:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757772.995444:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757772.995447:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757772.995452:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681664:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757772.995456:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995458:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757772.995460:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757772.995462:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757772.995464:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757772.995466:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757772.995468:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757772.995471:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757772.995473:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757772.995475:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757772.995477:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995479:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757772.995481:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757772.995483:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757772.995485:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757772.995486:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757772.995487:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757772.995491:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757772.995492:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757772.995495:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757772.995496:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995499:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995503:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757772.995505:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995509:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757772.995511:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757772.995513:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757772.995516:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757772.995518:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.995521:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757772.995524:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757772.995527:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757772.995534:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681792:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757772.995541:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757772.995545:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757772.995548:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757772.995552:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757772.995557:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995587:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757772.995591:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757772.995598:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757772.995624:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757772.995633:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757772.995638:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757772.995641:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995644:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757772.995650:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995657:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305500172 00000100:00000001:0.0:1787757772.995661:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305500865 00000100:00000001:0.0:1787757772.995664:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305500172 00000100:00000001:0.0:1787757772.995667:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305500865 00000100:00000001:0.0:1787757772.995670:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757772.995672:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995674:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757772.995680:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757772/real 1787757772] req@ffff8ffb47a39180 x1874588659681792/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757772.995690:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757772.995692:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757772.995696:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757772.995698:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757772.995700:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995704:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757772.995706:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757772.995710:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659681792/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757772.995720:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995723:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757772.995725:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995727:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757772.995729:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995731:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757772.995750:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757772.995754:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757772.995757:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757772.995760:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757772.995764:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659681792:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757772.995769:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995771:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757772.995773:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757772.995775:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757772.995778:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757772.995780:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757772.995781:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757772.995785:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757772.995787:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757772.995791:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757772.995793:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995795:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757772.995800:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757772.995802:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757772.995805:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757772.995807:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757773.569380:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757773.569393:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757773.569396:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757773.569405:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757773.569410:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757773.569416:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757773.569419:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757773.569422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757773.569426:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757773.761427:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757773.761439:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757773.761448:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757773.761449:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757773.761531:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757773.761541:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757773.761588:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757774.593360:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757774.593372:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757774.593376:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757774.593384:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757774.593389:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757774.593396:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757774.593399:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757774.593404:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757774.593408:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757774.785331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757774.785339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757774.785347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757774.785348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757774.785450:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757774.785457:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757774.785462:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757775.617287:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757775.617296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757775.617299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757775.617305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757775.617310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757775.617315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757775.617318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757775.617321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757775.617325:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757775.809300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757775.809310:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757775.809313:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757775.809317:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757775.809324:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757775.809328:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757775.809332:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757775.809335:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757775.809341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757775.809343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757775.809479:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757775.809485:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757775.809490:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757776.641304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757776.641312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757776.641316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757776.641322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757776.641327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757776.641331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757776.641334:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757776.641338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757776.641341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757776.833282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757776.833287:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757776.833292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757776.833293:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757776.833384:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757776.833388:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757776.833390:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757777.665318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757777.665327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757777.665329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757777.665336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757777.665341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757777.665346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757777.665349:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757777.665353:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757777.665356:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757777.857374:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757777.857380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757777.857385:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757777.857386:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757777.857470:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757777.857522:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757777.857526:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757778.113309:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757778.113321:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757778.113325:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757778.113329:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757778.113335:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757778.113344:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10832 00000100:00080000:2.0:1787757778.113358:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757778.113362:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757778.113365:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757778.113370:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757778.113373:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757778.113377:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757778.113380:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757778.113382:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757778.113384:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757778.113391:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757778.113394:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757778.113398:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757778.113403:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757778.113407:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113410:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757778.113413:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757778.113425:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757778.113427:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113432:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757778.113435:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757778.113440:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113454:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757778.113467:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659681920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757778.113531:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757778.113537:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113554:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757778.113559:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757778.113563:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757778.113565:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757778.113567:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757778.113571:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757778.113574:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10832 00000100:00080000:2.0:1787757778.113579:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757778.113581:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757778.113583:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757778.113586:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757778.113588:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757778.113591:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757778.113593:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757778.113595:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757778.113597:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757778.113601:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757778.113604:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757778.113608:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757778.113615:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757778.113619:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113621:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757778.113628:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757778.113633:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00000001:2.0:1787757778.113633:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113642:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.113644:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757778.113650:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757778.113651:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787757778.113657:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.113660:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757778.113663:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757778.113668:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757778.113670:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757778.113672:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757778.113686:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757778.113687:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757778.113695:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00100000:2.0:1787757778.113699:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659682048/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757778.113708:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113722:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757778.113727:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757778.113731:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659681920:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757778.113733:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757778.113741:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757778.113752:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787757778.113753:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757778.113761:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757778.113763:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757778.113765:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787757778.113768:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757778.113775:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757778.113776:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10832 02000000:00000001:0.0:1787757778.113788:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:2.0:1787757778.113788:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757778.113795:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757778.113797:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757778.113800:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757778.113802:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787757778.113803:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757778.113805:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757778.113815:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757778.113817:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757778.113820:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757778.113824:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757778.113827:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757778.113829:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757778.113833:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757778.113837:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.113838:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757778.113840:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757778.113847:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000008:00100000:2.0:1787757778.113848:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757778.113859:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113864:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.113865:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787757778.113871:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757778.113876:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113878:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757778.113880:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757778.113882:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113889:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757778.113907:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659682176/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757778.113918:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757778.113921:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.113924:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757778.113928:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757778.113936:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787757778.113939:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757778.113940:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757778.113946:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757778.113952:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757778.113957:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10832 00000100:00080000:2.0:1787757778.113964:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757778.113966:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757778.113967:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757778.113974:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757778.113978:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757778.113980:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:2.0:1787757778.113982:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757778.113988:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757778.113990:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757778.113992:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757778.114002:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757778.114003:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757778.114011:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757778.114016:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757778.114020:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114021:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757778.114023:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757778.114029:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757778.114034:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114036:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305505985 00000100:00000001:2.0:1787757778.114037:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757778.114042:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757778.114048:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.114050:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757778.114057:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757778.114059:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757778.114060:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.114069:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757778.114071:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757778.114073:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305505985 02000000:00000001:2.0:1787757778.114075:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757778.114084:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757778.114091:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659682304/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757778.114104:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757778.114105:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757778.114113:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114114:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757778.114118:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114122:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114129:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757778.114140:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757778/real 1787757778] req@ffff8ffb47a38700 x1874588659681920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757778.114157:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757778.114160:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757778.114166:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757778.114170:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757778.114172:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114175:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757778.114178:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757778.114223:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659681920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757778.114244:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114248:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757778.114252:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114254:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757778.114255:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114258:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757778.114261:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757778.114265:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757778.114270:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757778.114273:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757778.114279:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659681920:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757778.114285:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114289:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757778.114291:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757778.114294:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757778.114299:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757778.114301:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757778.114304:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757778.114310:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757778.114312:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757778.114316:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757778.114318:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114321:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114329:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757778.114331:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114334:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757778.114336:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757778.114339:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757778.114341:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757778.114343:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757778.114346:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757778.114349:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757778.114352:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757778.114357:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682048:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757778.114361:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757778.114363:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757778.114366:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757778.114368:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757778.114373:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114381:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757778.114383:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:0.0:1787757778.114389:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757778.114408:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757778.114415:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757778.114418:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757778.114420:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114422:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757778.114426:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114431:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305505294 00000100:00000001:0.0:1787757778.114435:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305505985 00000100:00000001:0.0:1787757778.114438:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305505294 00000100:00000001:0.0:1787757778.114441:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305505985 00000100:00000001:0.0:1787757778.114444:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757778.114446:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114449:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757778.114454:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757778/real 1787757778] req@ffff8ffb47a3a300 x1874588659682048/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757778.114464:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757778.114466:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757778.114501:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757778.114504:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757778.114506:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114508:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757778.114511:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757778.114517:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659682048/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757778.114535:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114538:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757778.114541:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114543:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757778.114545:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114547:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757778.114563:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757778.114566:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757778.114570:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757778.114572:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757778.114576:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682048:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757778.114581:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757778.114583:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757778.114585:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757778.114588:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757778.114591:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757778.114593:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757778.114597:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682176:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757778.114601:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757778.114604:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757778.114606:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757778.114608:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757778.114612:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114619:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757778.114621:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757778.114626:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757778.114642:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757778.114648:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757778.114651:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757778.114652:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114655:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757778.114659:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114663:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305505291 00000100:00000001:0.0:1787757778.114666:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305505985 00000100:00000001:0.0:1787757778.114669:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305505291 00000100:00000001:0.0:1787757778.114672:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305505985 00000100:00000001:0.0:1787757778.114675:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757778.114677:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114679:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757778.114684:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757778/real 1787757778] req@ffff8ffb47a39180 x1874588659682176/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757778.114693:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757778.114694:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757778.114697:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757778.114699:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757778.114701:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114703:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757778.114705:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757778.114709:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659682176/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757778.114717:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114720:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757778.114723:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114724:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757778.114726:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114728:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757778.114742:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757778.114744:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757778.114748:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757778.114750:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757778.114755:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682176:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757778.114759:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757778.114761:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757778.114763:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757778.114765:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757778.114767:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757778.114769:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757778.114773:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682304:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757778.114776:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757778.114778:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757778.114780:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757778.114782:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757778.114786:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114791:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757778.114792:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757778.114797:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757778.114808:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757778.114813:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757778.114816:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757778.114817:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114820:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757778.114823:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114826:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305505291 00000100:00000001:0.0:1787757778.114830:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305505985 00000100:00000001:0.0:1787757778.114832:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305505291 00000100:00000001:0.0:1787757778.114835:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305505985 00000100:00000001:0.0:1787757778.114838:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757778.114840:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.114842:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757778.114847:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757778/real 1787757778] req@ffff8ffb47a3b480 x1874588659682304/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757778.114864:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757778.114865:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757778.114868:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757778.114870:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757778.114872:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114874:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757778.114875:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757778.114880:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659682304/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757778.114887:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114890:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757778.114892:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114894:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757778.114896:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114898:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757778.114912:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757778.114914:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757778.114918:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757778.114922:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757778.114928:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682304:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757778.114935:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114939:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757778.114941:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757778.114944:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757778.114948:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757778.114951:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757778.114953:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757778.114957:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757778.114959:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757778.114964:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757778.114967:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.114971:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757778.114973:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757778.114976:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757778.114981:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757778.114984:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757778.114986:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757778.114989:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757778.114991:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757778.114996:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757778.114998:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.115002:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757778.115005:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757778.115007:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757778.115011:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757778.115014:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757778.115017:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757778.115020:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757778.115022:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757778.115026:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757778.115029:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.115032:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.115039:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757778.115041:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.115044:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757778.115046:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757778.689289:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757778.689301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757778.689306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757778.689315:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757778.689322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757778.689328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.689331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757778.689334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757778.689338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757778.881368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757778.881383:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757778.881395:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757778.881397:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757778.881520:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757778.881526:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757778.881531:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757779.713310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757779.713319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757779.713322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757779.713329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757779.713333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757779.713338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757779.713341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757779.713344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757779.713347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757779.905348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757779.905357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757779.905364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757779.905366:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757779.905459:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757779.905465:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757779.905470:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757780.737309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757780.737320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757780.737324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757780.737332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757780.737337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757780.737344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757780.737348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757780.737352:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757780.737356:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757780.929330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757780.929341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757780.929351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757780.929353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757780.929473:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757780.929479:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757780.929484:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757781.761303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757781.761313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757781.761316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757781.761322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757781.761327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757781.761332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757781.761335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757781.761339:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757781.761342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757781.953354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757781.953379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757781.953389:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757781.953392:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757781.953534:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757781.953544:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757781.953551:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757782.785365:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757782.785434:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757782.785440:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757782.785448:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757782.785455:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757782.785462:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757782.785467:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757782.785471:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757782.785476:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757782.977367:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757782.977375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757782.977382:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757782.977406:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757782.977409:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757782.977413:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757782.977417:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757782.977421:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757782.977423:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757782.977426:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757782.977471:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757782.977475:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757782.977478:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757783.233317:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757783.233362:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757783.233365:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757783.233369:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757783.233375:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757783.233401:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10837 00000100:00080000:3.0:1787757783.233415:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757783.233420:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757783.233422:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757783.233428:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757783.233430:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757783.233435:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757783.233438:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757783.233441:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757783.233443:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757783.233449:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757783.233453:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757783.233456:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757783.233460:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757783.233465:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233467:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757783.233470:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757783.233482:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757783.233485:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233489:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757783.233492:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757783.233516:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233529:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757783.233542:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659682432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757783.233556:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757783.233561:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233582:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757783.233587:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757783.233591:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757783.233593:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757783.233596:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757783.233600:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757783.233603:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10837 00000100:00080000:3.0:1787757783.233607:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757783.233609:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757783.233612:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757783.233616:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757783.233618:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757783.233621:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757783.233625:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757783.233628:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757783.233630:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757783.233635:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757783.233640:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757783.233642:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757783.233648:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757783.233648:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757783.233656:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233658:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757783.233659:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757783.233664:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:3.0:1787757783.233669:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757783.233670:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757783.233676:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.233682:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757783.233686:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787757783.233691:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.233693:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233695:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757783.233705:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:0.0:1787757783.233706:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659682432:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757783.233709:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233719:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757783.233721:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757783.233724:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757783.233729:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757783.233730:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.233736:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.233741:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757783.233748:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233769:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757783.233773:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787757783.233775:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233779:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:3.0:1787757783.233789:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659682560/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757783.233802:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757783.233806:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233813:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757783.233818:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757783.233822:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757783.233824:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757783.233827:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757783.233832:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757783.233836:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10837 00000100:00080000:3.0:1787757783.233841:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757783.233844:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757783.233846:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:0.0:1787757783.233852:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787757783.233854:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757783.233861:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757783.233864:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757783.233868:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757783.233872:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757783.233873:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757783.233874:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757783.233883:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757783.233885:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757783.233891:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233893:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233899:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757783.233900:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757783.233908:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757783.233909:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757783.233914:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233918:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305511105 00000100:00000001:3.0:1787757783.233920:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757783.233928:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757783.233933:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233936:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757783.233937:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233949:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305511105 02000000:00000001:3.0:1787757783.233950:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757783.233956:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233961:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757783.233963:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757783.233964:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757783.233972:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233973:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757783.233980:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757783.233982:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.233984:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757783.234000:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757783/real 1787757783] req@ffff8ffb45c31500 x1874588659682432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757783.234001:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659682688/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787757783.234058:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757783.234059:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:0.0:1787757783.234065:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757783.234067:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234072:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757783.234074:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234078:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787757783.234082:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757783.234084:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234093:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757783.234094:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757783.234097:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757783.234100:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757783.234105:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757783.234112:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659682432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757783.234112:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757783.234123:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10837 00000100:00080000:3.0:1787757783.234134:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757783.234139:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757783.234141:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757783.234146:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757783.234152:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234155:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757783.234157:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757783.234159:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757783.234167:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757783.234170:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000020:00000001:3.0:1787757783.234172:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757783.234179:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757783.234180:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787757783.234219:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757783.234222:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757783.234226:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757783.234231:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757783.234235:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757783.234238:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757783.234242:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757783.234254:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757783.234256:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757783.234258:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757783.234265:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757783.234273:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757783.234275:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757783.234277:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234288:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1787757783.234289:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757783.234295:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757783.234302:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659682432:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757783.234303:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757783.234307:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757783.234313:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234319:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757783.234321:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234325:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757783.234330:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757783.234335:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00100000:3.0:1787757783.234336:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659682816/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757783.234342:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757783.234349:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757783.234354:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757783.234355:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757783.234360:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787757783.234362:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.234366:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757783.234370:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234372:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757783.234376:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234403:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234413:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757783.234416:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234428:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757783.234431:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757783.234433:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757783.234435:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757783.234437:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.234441:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757783.234443:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757783.234446:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757783.234451:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682560:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757783.234456:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757783.234459:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757783.234461:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.234464:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757783.234468:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234477:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757783.234479:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757783.234486:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757783.234505:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757783.234512:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757783.234516:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757783.234517:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234519:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757783.234523:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234529:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305510414 00000100:00000001:0.0:1787757783.234533:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305511105 00000100:00000001:0.0:1787757783.234536:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305510414 00000100:00000001:0.0:1787757783.234539:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305511105 00000100:00000001:0.0:1787757783.234541:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757783.234544:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234546:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757783.234552:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757783/real 1787757783] req@ffff8ffb45c30700 x1874588659682560/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757783.234562:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757783.234564:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757783.234567:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757783.234569:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757783.234571:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234573:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757783.234575:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757783.234580:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659682560/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757783.234589:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234592:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757783.234595:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234597:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757783.234598:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234601:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757783.234616:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757783.234620:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757783.234624:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757783.234627:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757783.234631:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682560:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757783.234636:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757783.234637:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757783.234639:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.234642:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757783.234644:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757783.234646:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757783.234650:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682688:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757783.234654:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757783.234656:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757783.234658:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.234660:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757783.234663:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234669:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757783.234670:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757783.234675:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757783.234687:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757783.234691:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757783.234694:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757783.234696:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234698:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757783.234701:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234705:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305510411 00000100:00000001:0.0:1787757783.234708:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305511105 00000100:00000001:0.0:1787757783.234711:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305510411 00000100:00000001:0.0:1787757783.234714:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305511105 00000100:00000001:0.0:1787757783.234717:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757783.234719:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234722:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757783.234727:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757783/real 1787757783] req@ffff8ffb45c32a00 x1874588659682688/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757783.234737:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757783.234738:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757783.234740:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757783.234742:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757783.234745:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234747:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757783.234748:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757783.234753:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659682688/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757783.234761:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234763:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757783.234766:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234768:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757783.234769:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234771:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757783.234783:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757783.234786:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757783.234789:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757783.234792:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757783.234795:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682688:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757783.234799:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757783.234801:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757783.234803:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.234805:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757783.234807:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757783.234809:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757783.234813:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682816:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757783.234820:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757783.234822:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757783.234824:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757783.234826:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757783.234829:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234834:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757783.234836:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787757783.234840:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757783.234851:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757783.234856:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757783.234858:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757783.234860:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234862:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757783.234865:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234869:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305510411 00000100:00000001:0.0:1787757783.234872:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305511105 00000100:00000001:0.0:1787757783.234875:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305510411 00000100:00000001:0.0:1787757783.234878:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305511105 00000100:00000001:0.0:1787757783.234880:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757783.234882:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.234885:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757783.234889:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757783/real 1787757783] req@ffff8ffb45c33800 x1874588659682816/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757783.234898:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757783.234899:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757783.234901:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757783.234903:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757783.234905:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234907:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757783.234909:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757783.234913:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659682816/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757783.234930:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234933:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757783.234936:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234938:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757783.234939:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234941:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757783.234953:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757783.234956:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757783.234959:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757783.234962:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757783.234965:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659682816:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757783.234970:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234973:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757783.234974:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757783.234976:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757783.234979:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757783.234981:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757783.234983:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757783.234985:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757783.234987:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757783.234989:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757783.234991:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.234993:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757783.234995:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757783.234996:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757783.234999:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757783.235001:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757783.235002:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757783.235004:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757783.235005:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757783.235007:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757783.235009:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.235011:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757783.235012:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757783.235015:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757783.235017:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757783.235018:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757783.235020:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757783.235022:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757783.235023:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757783.235025:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757783.235027:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.235029:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.235033:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757783.235035:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.235037:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757783.235039:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757783.809335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757783.809347:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757783.809350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757783.809359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757783.809364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757783.809400:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.809405:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757783.809410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757783.809414:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757784.001297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757784.001306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757784.001313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757784.001315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757784.001358:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757784.001363:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757784.001389:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757784.833341:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757784.833378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757784.833381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757784.833389:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757784.833395:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757784.833401:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757784.833405:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757784.833410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757784.833414:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757785.025532:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757785.025550:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757785.025559:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757785.025561:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757785.025642:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757785.025651:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757785.025658:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757785.857299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757785.857309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757785.857311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757785.857318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757785.857323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757785.857328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757785.857331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757785.857358:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757785.857362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757786.049296:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757786.049310:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757786.049315:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757786.049320:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757786.049377:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757786.049382:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757786.049388:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757786.049393:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757786.049402:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757786.049406:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757786.049491:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757786.049497:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757786.049503:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757786.881338:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757786.881349:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757786.881353:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757786.881363:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757786.881369:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757786.881376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757786.881381:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757786.881385:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757786.881390:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757787.073357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757787.073365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757787.073373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757787.073375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757787.073402:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757787.073405:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757787.073409:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757787.905344:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757787.905357:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757787.905360:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757787.905368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757787.905373:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757787.905378:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757787.905382:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757787.905387:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757787.905391:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757788.097347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757788.097364:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757788.097379:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757788.097382:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757788.097431:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757788.097435:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757788.097440:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757788.353472:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757788.353487:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757788.353491:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757788.353497:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757788.353504:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757788.353516:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10843 00000100:00080000:2.0:1787757788.353534:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757788.353539:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757788.353542:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757788.353548:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757788.353552:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757788.353557:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757788.353560:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757788.353563:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757788.353565:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757788.353572:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757788.353575:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757788.353579:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757788.353585:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757788.353589:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353592:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757788.353595:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757788.353614:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757788.353616:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353621:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757788.353626:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757788.353633:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353653:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757788.353668:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659682944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757788.353684:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757788.353690:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353719:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757788.353723:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757788.353728:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757788.353730:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757788.353733:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757788.353737:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757788.353739:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10843 00000100:00080000:2.0:1787757788.353744:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757788.353746:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757788.353748:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757788.353751:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757788.353752:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757788.353755:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757788.353758:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757788.353760:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757788.353762:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757788.353765:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757788.353768:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757788.353770:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757788.353778:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757788.353780:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353782:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757788.353788:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757788.353792:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353795:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757788.353801:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757788.353803:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353805:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757788.353808:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757788.353810:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353818:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757788.353827:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659683072/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757788.353840:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757788.353844:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757788.353845:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.353867:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757788.353867:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757788.353878:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757788.353879:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757788.353888:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:2.0:1787757788.353892:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757788.353894:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.353897:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757788.353904:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757788.353914:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757788.353915:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:2.0:1787757788.353927:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10843 00000100:00080000:2.0:1787757788.353933:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787757788.353934:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757788.353936:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757788.353940:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757788.353950:0:87148:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:0.0:1787757788.353952:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757788.353954:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757788.353958:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757788.353964:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757788.353968:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757788.353971:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787757788.353978:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659682944:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787757788.353980:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757788.353988:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757788.354000:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757788.354013:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757788.354014:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757788.354021:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.354026:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757788.354030:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:2.0:1787757788.354032:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757788.354038:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757788.354040:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.354042:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757788.354057:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757788.354059:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757788.354061:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.354068:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757788.354070:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757788.354072:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757788.354077:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.354082:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757788.354094:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659683200/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757788.354104:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757788.354107:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:0.0:1787757788.354113:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757788.354122:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757788.354127:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.354131:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757788.354135:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757788.354140:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757788.354142:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757788.354144:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757788.354149:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757788.354154:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10843 00000100:00080000:2.0:1787757788.354160:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757788.354162:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757788.354165:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:0.0:1787757788.354175:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787757788.354177:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757788.354181:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757788.354246:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757788.354253:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757788.354262:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757788.354270:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757788.354273:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354277:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757788.354284:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354323:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305516226 00000100:00000001:0.0:1787757788.354339:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787757788.354349:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305516226 00000100:00000001:0.0:1787757788.354355:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757788.354369:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757788.354373:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354376:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757788.354387:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757788/real 1787757788] req@ffff8ffb47a38000 x1874588659682944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757788.354414:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:0.0:1787757788.354418:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757788.354419:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:0.0:1787757788.354429:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:2.0:1787757788.354436:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757788.354440:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757788.354443:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354446:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:2.0:1787757788.354450:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757788.354452:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.354460:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:2.0:1787757788.354461:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757788.354468:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757788.354471:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.354477:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787757788.354483:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659682944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757788.354485:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757788.354505:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757788.354517:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354535:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757788.354536:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757788.354545:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354547:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757788.354558:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757788.354559:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757788.354570:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757788.354572:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757788.354583:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757788.354584:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354597:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757788.354601:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757788.354603:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757788.354611:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757788.354621:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1787757788.354623:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659683328/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757788.354629:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757788.354650:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659682944:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757788.354670:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757788.354682:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354687:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757788.354691:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354695:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757788.354701:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757788.354707:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757788.354713:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757788.354719:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757788.354722:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757788.354728:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757788.354733:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757788.354740:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757788.354742:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.354746:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.354755:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757788.354757:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354761:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757788.354763:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757788.354765:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757788.354768:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757788.354770:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757788.354773:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757788.354776:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757788.354780:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757788.354785:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683072:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757788.354790:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757788.354793:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757788.354795:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757788.354797:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757788.354802:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354810:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757788.354812:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:0.0:1787757788.354818:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757788.354838:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757788.354844:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757788.354847:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757788.354849:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354851:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757788.354855:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354860:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305515534 00000100:00000001:0.0:1787757788.354864:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305516226 00000100:00000001:0.0:1787757788.354867:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305515534 00000100:00000001:0.0:1787757788.354870:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305516226 00000100:00000001:0.0:1787757788.354873:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757788.354876:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.354878:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757788.354883:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757788/real 1787757788] req@ffff8ffb47a38380 x1874588659683072/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757788.354894:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757788.354895:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757788.354898:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757788.354900:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757788.354902:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.354904:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757788.354906:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757788.354913:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659683072/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757788.354928:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.354932:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757788.354937:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.354940:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757788.354942:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.354945:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757788.354969:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757788.354975:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757788.354982:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757788.354988:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757788.354995:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683072:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757788.355007:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757788.355010:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757788.355013:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757788.355018:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757788.355021:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757788.355041:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757788.355047:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757788.355057:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757788.355060:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757788.355063:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757788.355065:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757788.355070:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355079:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757788.355081:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757788.355087:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757788.355105:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757788.355111:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757788.355114:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757788.355116:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355118:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757788.355122:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355126:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305515531 00000100:00000001:0.0:1787757788.355130:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305516226 00000100:00000001:0.0:1787757788.355133:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305515531 00000100:00000001:0.0:1787757788.355136:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305516226 00000100:00000001:0.0:1787757788.355138:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757788.355141:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355143:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757788.355148:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757788/real 1787757788] req@ffff8ffb47a38a80 x1874588659683200/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757788.355158:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757788.355160:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757788.355163:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757788.355165:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757788.355167:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355169:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757788.355171:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757788.355175:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659683200/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757788.355220:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355224:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757788.355227:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355229:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757788.355231:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355233:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757788.355247:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757788.355250:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757788.355254:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757788.355257:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757788.355261:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757788.355267:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757788.355269:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757788.355271:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757788.355274:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757788.355277:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757788.355280:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757788.355288:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683328:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757788.355317:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757788.355320:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757788.355323:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757788.355326:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757788.355333:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355341:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757788.355344:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757788.355349:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757788.355367:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757788.355374:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757788.355377:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757788.355379:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355381:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757788.355384:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355389:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305515532 00000100:00000001:0.0:1787757788.355392:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305516226 00000100:00000001:0.0:1787757788.355395:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305515532 00000100:00000001:0.0:1787757788.355398:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305516226 00000100:00000001:0.0:1787757788.355401:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757788.355403:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355406:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757788.355411:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757788/real 1787757788] req@ffff8ffb47a3a680 x1874588659683328/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757788.355420:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757788.355421:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757788.355424:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757788.355426:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757788.355428:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355430:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757788.355433:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757788.355437:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659683328/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757788.355445:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355448:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757788.355451:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355453:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757788.355454:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355456:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757788.355470:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757788.355473:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757788.355477:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757788.355479:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757788.355483:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683328:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757788.355487:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355490:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757788.355491:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757788.355493:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757788.355496:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757788.355498:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757788.355500:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757788.355503:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757788.355505:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757788.355508:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757788.355509:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355512:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757788.355513:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757788.355515:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757788.355518:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757788.355519:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757788.355521:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757788.355523:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757788.355524:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757788.355525:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757788.355527:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355529:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757788.355530:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757788.355531:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757788.355534:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757788.355535:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757788.355536:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757788.355539:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757788.355541:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757788.355544:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757788.355545:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355547:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.355552:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757788.355554:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.355557:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757788.355559:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757788.929324:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757788.929336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757788.929339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757788.929346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757788.929352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757788.929358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.929362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757788.929366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757788.929371:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757789.121337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757789.121346:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757789.121354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757789.121358:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757789.121419:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757789.121426:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757789.121432:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757789.953330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757789.953343:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757789.953348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757789.953357:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757789.953363:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757789.953369:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757789.953373:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757789.953377:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757789.953381:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757790.145367:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757790.145375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757790.145382:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757790.145386:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757790.145461:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757790.145470:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757790.145477:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757790.977295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757790.977305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757790.977308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757790.977314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757790.977319:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757790.977323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757790.977327:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757790.977330:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757790.977334:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757791.169365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757791.169375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757791.169386:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757791.169388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757791.169432:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757791.169436:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757791.169440:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757792.001322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757792.001331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757792.001333:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757792.001340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757792.001345:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757792.001350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757792.001353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757792.001356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757792.001360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757792.193341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757792.193350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757792.193361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757792.193368:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757792.193373:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757792.193377:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757792.193382:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757792.193387:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757792.193389:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757792.193395:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757792.193425:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757792.193429:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757792.193432:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757793.025364:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757793.025375:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757793.025379:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757793.025387:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757793.025393:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757793.025398:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757793.025402:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757793.025406:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757793.025410:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757793.217331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757793.217340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757793.217349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757793.217351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757793.217454:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757793.217460:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757793.217464:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757793.473374:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757793.473387:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757793.473390:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757793.473395:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757793.473401:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757793.473409:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10848 00000100:00080000:3.0:1787757793.473424:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757793.473428:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757793.473431:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757793.473437:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757793.473440:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757793.473444:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757793.473447:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757793.473450:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757793.473452:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757793.473458:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757793.473461:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.473465:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757793.473469:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757793.473473:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473476:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757793.473479:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.473491:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757793.473494:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473497:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757793.473501:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757793.473506:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473520:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757793.473533:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659683456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757793.473546:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757793.473552:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473575:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757793.473579:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757793.473584:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757793.473586:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757793.473588:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757793.473592:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757793.473596:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10848 00000100:00080000:3.0:1787757793.473601:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757793.473603:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757793.473606:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757793.473609:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757793.473610:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757793.473613:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757793.473615:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757793.473618:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757793.473619:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757793.473623:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757793.473625:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.473627:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757793.473633:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757793.473636:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473638:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757793.473644:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757793.473649:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473652:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.473657:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757793.473659:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473662:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757793.473663:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757793.473666:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473673:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757793.473681:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659683584/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757793.473682:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757793.473691:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757793.473698:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757793.473699:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757793.473703:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757793.473705:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.473709:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473713:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757793.473719:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757793.473721:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757793.473727:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757793.473730:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757793.473736:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473738:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757793.473746:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757793.473752:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659683456:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757793.473754:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757793.473765:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10848 00000100:00000001:0.0:1787757793.473768:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757793.473772:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787757793.473776:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757793.473778:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757793.473783:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473784:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757793.473792:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787757793.473793:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757793.473799:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757793.473802:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757793.473803:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757793.473806:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757793.473812:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757793.473826:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757793.473832:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757793.473833:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787757793.473844:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757793.473851:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757793.473859:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.473863:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757793.473870:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757793.473875:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473880:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757793.473885:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757793.473890:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473895:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.473905:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757793.473908:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473912:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757793.473915:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757793.473918:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.473921:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.473928:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305521345 00000100:00000001:3.0:1787757793.473937:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.473945:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:3.0:1787757793.473956:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659683712/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757793.473957:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305521345 00000100:00000001:3.0:1787757793.473976:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757793.473980:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.473985:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.473990:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787757793.473991:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757793.474003:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757793.474006:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757793.474009:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757793.474010:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757793.474019:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757793.474023:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757793.474027:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10848 00000100:00000001:0.0:1787757793.474037:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757793.474040:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757793.474049:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757793.474051:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757793.474052:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757793.474064:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757793.474066:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757793.474069:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757793.474073:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757793.474074:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757793.474081:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757793.474084:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:3.0:1787757793.474086:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757793.474098:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787757793.474099:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757793.474107:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757793.474108:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474115:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757793.474116:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757793.474122:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757793.474126:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757793.474128:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.474137:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757793.474139:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757793.474143:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757793.474146:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:3.0:1787757793.474149:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757793.474153:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757793.474157:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757793.474166:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683584:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757793.474167:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.474182:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757793.474246:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757793.474253:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757793.474256:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757793.474263:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757793.474263:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.474272:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757793.474274:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757793.474275:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757793.474283:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474292:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757793.474297:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474298:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:0.0:1787757793.474312:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:3.0:1787757793.474317:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659683840/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757793.474331:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757793.474335:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474338:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757793.474341:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474347:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305520654 00000100:00000001:0.0:1787757793.474355:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305521345 00000100:00000001:0.0:1787757793.474359:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305520654 00000100:00000001:0.0:1787757793.474362:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305521345 00000100:00000001:0.0:1787757793.474366:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757793.474369:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474374:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757793.474376:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757793.474378:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757793.474381:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757793.474384:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757793.474387:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757793.474394:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757793.474398:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757793.474400:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757793.474402:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757793.474404:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757793.474407:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474413:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757793.474415:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757793.474419:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787757793.474428:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474432:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305520651 00000100:00000001:0.0:1787757793.474435:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305521345 00000100:00000001:0.0:1787757793.474438:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305520651 00000100:00000001:0.0:1787757793.474441:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305521345 00000100:00000001:0.0:1787757793.474443:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757793.474445:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474448:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474450:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757793.474454:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757793.474456:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787757793.474459:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757793.474461:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757793.474462:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757793.474464:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757793.474465:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757793.474468:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757793.474470:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757793.474472:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757793.474476:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683840:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757793.474482:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757793.474484:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757793.474487:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757793.474490:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757793.474495:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474503:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757793.474506:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787757793.474512:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787757793.474531:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474537:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305520651 00000100:00000001:0.0:1787757793.474542:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305521345 00000100:00000001:0.0:1787757793.474550:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305520651 00000100:00000001:0.0:1787757793.474554:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305521345 00000100:00000001:0.0:1787757793.474558:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757793.474562:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474567:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474571:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757793.474577:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757793.474580:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787757793.474584:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757793.474586:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757793.474589:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757793.474594:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757794.049375:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757794.049385:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757794.049389:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757794.049395:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757794.049400:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757794.049405:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757794.049409:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757794.049412:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757794.049417:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757794.241316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757794.241324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757794.241334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757794.241337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757794.241384:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757794.241388:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757794.241394:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757795.073319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757795.073327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757795.073330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757795.073337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757795.073342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757795.073346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757795.073351:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757795.073354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757795.073357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757795.265316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757795.265331:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757795.265336:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757795.265343:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757795.265352:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757795.265356:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757795.265360:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757795.265364:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757795.265372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757795.265374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757795.265400:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757795.265403:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757795.265407:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757796.097273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757796.097280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757796.097282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757796.097287:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757796.097291:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757796.097295:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757796.097298:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757796.097301:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757796.097303:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757796.289298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757796.289304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757796.289311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757796.289314:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757796.289338:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757796.289340:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757796.289343:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757796.545539:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1787757796.545601:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757796.545622:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757796.545639:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757796.545667:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757796.545672:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757796.545677:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787757796.545692:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757796.545702:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757796.545707:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757796.545710:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757796.545713:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787757796.545721:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757796.545729:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757796.545733:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757796.545736:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757796.545739:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787757796.545747:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:0.0:1787757796.545751:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757796.545755:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757796.545759:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757796.545762:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757796.545765:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757796.545767:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757796.545772:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757796.545789:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757793/real 1787757796] req@ffff8ffb45c33800 x1874588659683840/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757796.545815:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757796.545819:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757796.545829:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757796.545834:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757796.545837:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.545842:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757796.545846:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757796.545855:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659683840/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757796.545870:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.545874:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757796.545879:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.545881:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757796.545883:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.545886:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757796.545902:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757796.545906:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757796.545913:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757796.545916:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757796.545922:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683840:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757796.545929:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757796.545934:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757793/real 1787757796] req@ffff8ffb45c33480 x1874588659683584/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757796.545955:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757796.545957:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757796.545961:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757796.545963:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757796.545974:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.545978:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757796.545980:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757796.545989:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659683584/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757796.546003:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546007:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757796.546011:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546014:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757796.546017:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546020:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757796.546041:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757796.546048:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757796.546055:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757796.546060:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757796.546068:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683584:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757796.546076:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757796.546094:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757793/real 1787757796] req@ffff8ffb45c30000 x1874588659683712/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757796.546111:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757796.546115:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757796.546121:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757796.546125:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757796.546128:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546131:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757796.546135:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757796.546143:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659683712/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757796.546218:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546224:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757796.546229:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546233:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757796.546235:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546239:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757796.546259:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757796.546265:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757796.546272:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757796.546276:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757796.546283:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659683712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757796.546291:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757796.546299:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757793/real 1787757796] req@ffff8ffb45c31180 x1874588659683456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757796.546317:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757796.546320:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757796.546327:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757796.546331:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757796.546334:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546337:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757796.546340:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757796.546347:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659683456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757796.546363:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546367:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757796.546372:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546376:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757796.546379:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546382:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757796.546385:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757796.546390:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757796.546397:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757796.546402:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757796.546408:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659683456:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757796.546417:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546421:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757796.546424:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757796.546429:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757796.546450:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757796.546455:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757796.546459:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757796.546466:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757796.546470:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757796.546475:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757796.546478:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546482:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757796.546484:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757796.546487:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757796.546492:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757796.546494:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757796.546497:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757796.546501:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757796.546503:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757796.546506:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757796.546508:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546512:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757796.546514:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757796.546516:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757796.546519:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757796.546521:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757796.546524:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757796.546527:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757796.546529:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757796.546532:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757796.546536:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757796.546542:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757796.546544:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757796.546548:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757796.546551:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757796.546553:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757796.546555:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757796.546557:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757796.546560:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757796.546563:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546567:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757796.546579:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757796.546583:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757796.546588:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757796.546591:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757797.121286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757797.121295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757797.121298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757797.121305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757797.121309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757797.121314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757797.121316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757797.121319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757797.121323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757797.313353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757797.313361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757797.313368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757797.313370:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757797.313461:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757797.313467:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757797.313472:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757798.145284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757798.145292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757798.145295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757798.145301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757798.145305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757798.145310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757798.145313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757798.145317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757798.145321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757798.337277:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757798.337289:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757798.337300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757798.337302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757798.337342:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757798.337346:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757798.337351:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757798.593308:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757798.593321:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757798.593325:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757798.593329:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757798.593336:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757798.593347:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10853 00000100:00080000:3.0:1787757798.593361:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757798.593365:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757798.593367:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757798.593373:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757798.593376:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757798.593380:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757798.593384:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757798.593386:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757798.593388:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757798.593394:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757798.593397:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757798.593401:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757798.593404:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757798.593410:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593414:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757798.593418:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757798.593434:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757798.593437:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593443:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757798.593447:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757798.593455:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593472:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757798.593490:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659683968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757798.593512:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757798.593520:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593548:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757798.593553:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757798.593557:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757798.593559:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757798.593562:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757798.593566:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757798.593570:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10853 00000100:00080000:3.0:1787757798.593575:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757798.593578:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757798.593581:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757798.593584:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757798.593586:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757798.593589:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757798.593592:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757798.593595:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757798.593597:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757798.593601:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757798.593605:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757798.593607:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757798.593614:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757798.593618:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593620:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757798.593627:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757798.593631:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593633:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.593637:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757798.593640:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757798.593646:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757798.593647:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.593655:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757798.593656:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757798.593662:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757798.593662:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757798.593665:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.593668:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.593678:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757798.593679:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.593683:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757798.593689:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757798.593693:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659684096/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757798.593702:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659683968:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757798.593715:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757798.593722:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757798.593727:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593732:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757798.593734:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757798.593746:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.593747:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593759:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757798.593769:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757798.593776:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757798.593779:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757798.593781:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757798.593789:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00080000:3.0:1787757798.593791:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757798.593805:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757798.593806:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757798.593819:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10853 00000100:00080000:3.0:1787757798.593828:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757798.593833:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757798.593836:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757798.593841:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757798.593844:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757798.593848:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757798.593854:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757798.593859:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757798.593861:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757798.593869:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757798.593879:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757798.593882:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757798.593889:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757798.593900:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757798.593905:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593909:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757798.593916:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757798.593920:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00000001:3.0:1787757798.593924:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.593931:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757798.593933:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.593940:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757798.593947:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757798.593949:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757798.593956:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593965:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757798.593966:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757798.593970:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757798.593980:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305526465 02000000:00000001:3.0:1787757798.593981:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.593998:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594006:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:3.0:1787757798.594017:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659684224/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757798.594033:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305526465 00000100:00000001:3.0:1787757798.594041:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757798.594051:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594052:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757798.594059:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594064:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757798.594069:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757798.594075:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757798.594083:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757798.594087:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757798.594090:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757798.594098:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757798.594138:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757798/real 1787757798] req@ffff8ffb45c32a00 x1874588659683968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757798.594141:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757798.594155:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10853 00000100:00080000:3.0:1787757798.594171:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757798.594181:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:0.0:1787757798.594228:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757798.594235:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757798.594245:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757798.594251:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757798.594255:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594261:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757798.594265:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757798.594276:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659683968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757798.594293:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594299:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757798.594305:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594308:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757798.594312:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594316:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757798.594320:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757798.594327:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757798.594334:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757798.594339:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757798.594348:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659683968:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757798.594358:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594364:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757798.594367:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757798.594372:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787757798.594382:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757798.594387:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757798.594395:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787757798.594397:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757798.594405:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787757798.594406:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757798.594414:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757798.594422:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000020:00000001:3.0:1787757798.594425:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757798.594430:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000020:00000001:3.0:1787757798.594438:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757798.594439:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757798.594452:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757798.594453:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757798.594462:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757798.594472:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:3.0:1787757798.594478:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757798.594480:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594495:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757798.594495:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594501:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757798.594505:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757798.594509:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757798.594516:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757798.594518:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757798.594520:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757798.594532:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.594534:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757798.594542:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757798.594545:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757798.594555:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757798.594556:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757798.594570:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757798.594577:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684096:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757798.594581:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594597:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757798.594600:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757798.594604:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757798.594608:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.594611:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.594618:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757798.594621:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757798.594626:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757798.594633:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757798.594636:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594652:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757798.594657:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757798.594671:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757798.594676:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757798.594695:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659684352/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757798.594712:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757798.594727:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757798.594728:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757798.594737:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757798.594739:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594746:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757798.594753:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594756:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757798.594769:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594779:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305525774 00000100:00000001:0.0:1787757798.594786:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305526465 00000100:00000001:0.0:1787757798.594791:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305525774 00000100:00000001:0.0:1787757798.594796:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305526465 00000100:00000001:0.0:1787757798.594801:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757798.594806:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.594809:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757798.594820:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757798/real 1787757798] req@ffff8ffb45c30700 x1874588659684096/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757798.594837:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757798.594840:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757798.594847:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757798.594850:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757798.594853:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594856:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757798.594858:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757798.594866:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659684096/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757798.594880:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594885:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757798.594890:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594893:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757798.594896:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.594900:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757798.594923:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757798.594929:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757798.594945:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757798.594950:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757798.594957:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684096:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757798.594965:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757798.594967:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757798.594970:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.594974:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757798.594978:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757798.594982:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757798.594987:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757798.594993:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757798.594997:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757798.595000:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.595004:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757798.595010:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595020:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757798.595023:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787757798.595030:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757798.595057:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757798.595067:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757798.595071:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757798.595074:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595078:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757798.595084:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595091:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305525771 00000100:00000001:0.0:1787757798.595096:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305526465 00000100:00000001:0.0:1787757798.595102:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305525771 00000100:00000001:0.0:1787757798.595106:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305526465 00000100:00000001:0.0:1787757798.595140:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757798.595145:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595149:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757798.595158:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757798/real 1787757798] req@ffff8ffb45c31500 x1874588659684224/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757798.595175:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757798.595177:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757798.595229:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757798.595234:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757798.595237:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595241:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757798.595244:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757798.595253:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659684224/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757798.595268:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595273:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757798.595276:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595278:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757798.595280:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595282:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757798.595296:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757798.595300:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757798.595304:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757798.595308:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757798.595313:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757798.595317:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595321:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757798.595323:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757798.595325:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757798.595330:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757798.595332:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757798.595335:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757798.595339:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757798.595342:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757798.595346:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757798.595349:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595352:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757798.595354:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757798.595357:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757798.595360:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757798.595362:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757798.595365:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757798.595369:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757798.595371:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757798.595373:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757798.595376:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595380:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595387:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757798.595390:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595394:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757798.595397:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757798.595400:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757798.595403:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757798.595406:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.595410:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757798.595414:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757798.595418:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757798.595426:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684352:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757798.595433:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757798.595437:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757798.595440:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757798.595443:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757798.595449:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595461:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757798.595465:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787757798.595471:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757798.595497:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757798.595505:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757798.595509:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757798.595511:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595514:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757798.595518:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595523:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305525772 00000100:00000001:0.0:1787757798.595527:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305526465 00000100:00000001:0.0:1787757798.595532:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305525772 00000100:00000001:0.0:1787757798.595536:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305526465 00000100:00000001:0.0:1787757798.595540:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757798.595544:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595548:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757798.595557:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757798/real 1787757798] req@ffff8ffb45c30a80 x1874588659684352/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757798.595573:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757798.595576:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757798.595582:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757798.595585:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757798.595588:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595592:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757798.595594:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757798.595602:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659684352/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757798.595616:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595621:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757798.595626:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595629:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757798.595632:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595635:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757798.595656:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757798.595660:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757798.595665:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757798.595668:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757798.595674:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684352:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757798.595679:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595681:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757798.595683:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757798.595685:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757798.595688:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757798.595690:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757798.595692:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757798.595696:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757798.595697:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757798.595700:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757798.595701:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595704:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757798.595709:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757798.595711:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757798.595713:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757798.595716:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757799.169330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757799.169347:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757799.169351:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757799.169361:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757799.169367:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757799.169372:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757799.169376:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757799.169381:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757799.169385:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757799.361332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757799.361342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757799.361353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757799.361356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757799.361419:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757799.361428:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757799.361434:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757800.193288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757800.193301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757800.193306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757800.193315:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757800.193323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757800.193329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757800.193334:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757800.193340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757800.193344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757800.385332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757800.385341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757800.385348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757800.385350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757800.385374:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757800.385378:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757800.385381:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757801.217279:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757801.217288:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757801.217291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757801.217298:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757801.217302:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757801.217307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757801.217311:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757801.217314:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757801.217318:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757801.409291:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757801.409300:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757801.409309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757801.409312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757801.409452:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757801.409458:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757801.409463:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757802.241298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757802.241308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757802.241311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757802.241318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757802.241322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757802.241327:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757802.241331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757802.241334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757802.241337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757802.433327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757802.433337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757802.433349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757802.433355:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757802.433360:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757802.433367:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757802.433373:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757802.433377:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757802.433381:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757802.433385:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757802.433425:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757802.433430:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757802.433435:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757803.265319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757803.265328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757803.265332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757803.265341:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757803.265347:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757803.265354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757803.265359:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757803.265364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757803.265369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757803.457318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757803.457331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757803.457342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757803.457344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757803.457439:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757803.457447:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757803.457454:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757803.713349:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757803.713364:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757803.713369:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757803.713374:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757803.713383:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757803.713395:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10858 00000100:00080000:2.0:1787757803.713411:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757803.713415:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757803.713418:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757803.713423:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757803.713427:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757803.713447:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757803.713453:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757803.713457:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757803.713460:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757803.713470:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757803.713476:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757803.713481:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757803.713488:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757803.713494:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713499:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757803.713504:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757803.713523:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757803.713526:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713530:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757803.713534:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757803.713539:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713554:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757803.713566:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659684480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757803.713581:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757803.713587:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713612:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757803.713617:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757803.713621:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757803.713623:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757803.713627:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757803.713631:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757803.713634:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10858 00000100:00080000:2.0:1787757803.713641:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757803.713644:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757803.713646:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757803.713650:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757803.713652:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757803.713656:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757803.713661:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757803.713665:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757803.713667:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757803.713674:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757803.713679:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757803.713682:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757803.713692:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757803.713698:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713703:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757803.713711:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757803.713719:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713726:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.713728:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757803.713736:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757803.713744:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757803.713745:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713754:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757803.713756:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787757803.713758:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757803.713773:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787757803.713773:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.713787:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713791:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757803.713812:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:2.0:1787757803.713813:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659684608/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757803.713830:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757803.713836:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757803.713849:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713849:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713877:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757803.713888:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659684480:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787757803.713889:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757803.713912:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757803.713917:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757803.713918:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757803.713924:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757803.713928:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787757803.713930:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757803.713935:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757803.713941:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10858 02000000:00000001:0.0:1787757803.713945:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:2.0:1787757803.713951:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757803.713958:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757803.713960:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.713962:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757803.713971:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757803.713973:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757803.713976:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757803.713980:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757803.713984:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757803.713987:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757803.713992:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757803.713994:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757803.713997:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.713999:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 02000000:00000001:2.0:1787757803.714003:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757803.714013:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787757803.714017:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757803.714061:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714066:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757803.714071:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757803.714077:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714082:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757803.714093:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757803.714096:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714100:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757803.714105:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757803.714109:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714122:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757803.714133:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659684736/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757803.714149:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757803.714171:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757803.714176:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714180:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757803.714239:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757803.714244:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757803.714246:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757803.714249:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757803.714254:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757803.714258:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10858 00000100:00080000:2.0:1787757803.714264:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757803.714269:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757803.714270:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757803.714282:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757803.714288:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757803.714291:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:2.0:1787757803.714292:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757803.714305:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757803.714307:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757803.714314:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757803.714318:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757803.714327:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757803.714332:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757803.714336:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757803.714345:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757803.714353:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305531586 00000100:00000001:2.0:1787757803.714355:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757803.714366:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757803.714374:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757803.714381:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757803.714382:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714396:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757803.714402:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757803.714404:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305531586 00000008:00000001:2.0:1787757803.714413:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714418:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.714421:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757803.714428:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757803.714433:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757803.714436:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.714448:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714449:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757803.714454:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757803.714460:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757803.714462:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714479:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757803.714482:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757803/real 1787757803] req@ffff8ffb47a39500 x1874588659684480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757803.714501:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659684864/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787757803.714525:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757803.714529:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:0.0:1787757803.714534:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757803.714536:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757803.714546:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.714550:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757803.714562:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757803.714567:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714573:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757803.714579:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757803.714591:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659684480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757803.714603:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714608:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757803.714613:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714615:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757803.714617:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714620:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757803.714623:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757803.714628:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757803.714633:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757803.714636:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757803.714642:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659684480:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757803.714649:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714653:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757803.714655:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757803.714659:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757803.714666:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757803.714670:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757803.714674:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757803.714680:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757803.714683:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757803.714689:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757803.714690:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714694:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714704:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757803.714707:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.714710:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757803.714713:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757803.714715:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757803.714718:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757803.714720:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757803.714724:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757803.714728:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757803.714733:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757803.714740:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684608:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757803.714748:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757803.714752:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757803.714755:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757803.714758:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757803.714764:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.714777:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757803.714781:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:0.0:1787757803.714789:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757803.714819:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757803.714830:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757803.714834:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757803.714838:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.714841:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757803.714847:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.714855:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305530894 00000100:00000001:0.0:1787757803.714860:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305531586 00000100:00000001:0.0:1787757803.714863:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305530894 00000100:00000001:0.0:1787757803.714866:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305531586 00000100:00000001:0.0:1787757803.714869:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757803.714872:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.714875:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757803.714881:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757803/real 1787757803] req@ffff8ffb47a39880 x1874588659684608/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757803.714894:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757803.714896:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757803.714899:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757803.714902:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757803.714904:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714906:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757803.714908:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757803.714913:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659684608/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757803.714923:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714925:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757803.714928:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714930:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757803.714932:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.714934:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757803.714957:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757803.714960:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757803.714964:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757803.714966:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757803.714970:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684608:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757803.714975:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757803.714978:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757803.714980:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757803.714983:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757803.714985:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757803.714988:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757803.714993:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684736:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757803.714998:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757803.715000:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757803.715003:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757803.715005:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757803.715010:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715017:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757803.715051:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757803.715058:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757803.715078:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757803.715085:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757803.715088:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757803.715090:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715092:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757803.715096:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715101:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305530891 00000100:00000001:0.0:1787757803.715104:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305531586 00000100:00000001:0.0:1787757803.715107:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305530891 00000100:00000001:0.0:1787757803.715111:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305531586 00000100:00000001:0.0:1787757803.715114:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757803.715116:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715119:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757803.715124:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757803/real 1787757803] req@ffff8ffb47a3a680 x1874588659684736/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757803.715133:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757803.715135:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757803.715138:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757803.715140:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757803.715142:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715144:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757803.715146:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757803.715151:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659684736/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757803.715169:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715172:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757803.715176:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715177:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757803.715179:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715181:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757803.715231:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757803.715235:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757803.715239:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757803.715241:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757803.715245:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684736:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757803.715250:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757803.715253:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757803.715255:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757803.715258:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757803.715260:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757803.715263:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757803.715267:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684864:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757803.715271:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757803.715274:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757803.715276:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757803.715278:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757803.715282:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715289:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757803.715291:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757803.715297:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757803.715311:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757803.715317:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757803.715319:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757803.715321:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715324:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757803.715328:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715332:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305530892 00000100:00000001:0.0:1787757803.715336:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305531586 00000100:00000001:0.0:1787757803.715339:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305530892 00000100:00000001:0.0:1787757803.715342:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305531586 00000100:00000001:0.0:1787757803.715345:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757803.715347:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715349:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757803.715354:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757803/real 1787757803] req@ffff8ffb47a38a80 x1874588659684864/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757803.715363:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757803.715364:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757803.715367:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757803.715369:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757803.715371:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715373:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757803.715375:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757803.715380:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659684864/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757803.715388:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715391:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757803.715393:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715395:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757803.715397:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715399:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757803.715417:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757803.715420:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757803.715424:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757803.715427:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757803.715431:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659684864:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757803.715436:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715439:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757803.715441:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757803.715443:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757803.715445:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757803.715447:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757803.715448:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757803.715453:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757803.715455:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757803.715458:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757803.715459:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715462:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757803.715463:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757803.715465:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757803.715468:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757803.715470:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757803.715471:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757803.715473:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757803.715474:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757803.715478:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757803.715480:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715482:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757803.715483:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757803.715485:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757803.715488:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757803.715490:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757803.715491:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757803.715494:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757803.715495:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757803.715499:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757803.715501:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715505:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757803.715511:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757803.715514:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757803.715518:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757803.715521:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757804.289305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757804.289314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757804.289316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757804.289323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757804.289329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757804.289333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757804.289337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757804.289341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757804.289344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757804.481331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757804.481342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757804.481352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757804.481356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757804.481388:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757804.481393:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757804.481397:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757805.313328:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757805.313338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757805.313341:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757805.313348:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757805.313352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757805.313357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757805.313360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757805.313363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757805.313366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757805.505292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757805.505303:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757805.505307:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757805.505311:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757805.505318:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757805.505321:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757805.505325:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757805.505329:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757805.505334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757805.505336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757805.505488:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757805.505494:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757805.505505:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757806.337330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757806.337339:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757806.337342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757806.337348:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757806.337353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757806.337358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757806.337362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757806.337366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757806.337369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757806.529314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757806.529322:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757806.529331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757806.529334:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757806.529360:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757806.529364:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757806.529367:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757807.361300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757807.361309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757807.361312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757807.361319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757807.361323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757807.361327:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757807.361330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757807.361333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757807.361337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757807.553328:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757807.553343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757807.553357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757807.553360:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757807.553522:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757807.553530:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757807.553535:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757808.385297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757808.385306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757808.385309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757808.385316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757808.385320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757808.385325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757808.385328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757808.385332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757808.385335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757808.577333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757808.577341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757808.577348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757808.577350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757808.577376:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757808.577379:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757808.577383:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757808.833426:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757808.833438:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757808.833442:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757808.833447:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757808.833453:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757808.833463:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10863 00000100:00080000:3.0:1787757808.833478:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757808.833483:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757808.833486:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757808.833492:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757808.833495:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757808.833500:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757808.833503:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757808.833506:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757808.833509:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757808.833518:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757808.833522:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.833526:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757808.833530:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757808.833535:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.833539:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757808.833542:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.833556:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757808.833558:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.833562:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757808.833566:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757808.833571:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.833585:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757808.833598:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659684992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757808.833612:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757808.833618:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.833646:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757808.833651:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757808.833656:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757808.833659:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757808.833662:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757808.833667:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757808.833673:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10863 00000100:00080000:3.0:1787757808.833681:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757808.833686:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757808.833690:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757808.833695:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757808.833698:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757808.833702:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757808.833706:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757808.833714:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757808.833716:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757808.833720:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757808.833729:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787757808.833731:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757808.833743:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757808.833743:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.833752:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757808.833754:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757808.833760:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.833762:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757808.833782:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757808.833785:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757808.833788:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:2.0:1787757808.833796:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757808.833803:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757808.833818:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659684992:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757808.833828:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757808.833836:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757808.833841:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757808.833846:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757808.833856:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.833886:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757808.833890:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:2.0:1787757808.833897:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00100000:3.0:1787757808.833902:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757808.833909:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.833913:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.833920:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757808.833923:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.833926:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757808.833955:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757808.833960:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.833969:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757808.833978:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659685120/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757808.833990:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757808.833994:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.833999:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757808.834003:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757808.834010:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787757808.834015:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757808.834016:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757808.834028:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757808.834034:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757808.834042:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10863 00000100:00000001:2.0:1787757808.834046:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757808.834050:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757808.834057:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757808.834057:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757808.834076:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834082:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757808.834091:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834097:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305536705 00000100:00000001:2.0:1787757808.834112:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757808.834120:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305536705 00000100:00000001:2.0:1787757808.834126:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757808.834131:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757808.834135:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834139:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757808.834151:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757808/real 1787757808] req@ffff8ffb45c33b80 x1874588659684992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757808.834168:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757808.834171:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757808.834176:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757808.834179:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757808.834232:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.834237:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757808.834241:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757808.834249:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659684992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757808.834261:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.834266:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757808.834272:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.834274:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757808.834277:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.834281:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757808.834285:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757808.834291:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757808.834298:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757808.834304:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757808.834311:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659684992:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757808.834316:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757808.834320:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757808.834325:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757808.834329:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:3.0:1787757808.834331:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787757808.834334:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000020:00000001:3.0:1787757808.834337:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787757808.834340:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000020:00000001:3.0:1787757808.834345:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757808.834353:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787757808.834355:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757808.834357:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757808.834370:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00080000:3.0:1787757808.834373:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000020:00000001:2.0:1787757808.834382:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757808.834382:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.834393:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757808.834399:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757808.834407:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757808.834416:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757808.834420:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.834426:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.834446:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757808.834451:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834459:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757808.834462:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757808.834465:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757808.834467:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757808.834469:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757808.834473:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757808.834477:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757808.834483:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757808.834494:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685120:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757808.834501:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757808.834509:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757808.834511:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757808.834515:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757808.834521:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.834525:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757808.834534:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834537:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757808.834538:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757808.834548:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000008:00100000:3.0:1787757808.834552:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757808.834562:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834564:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757808.834577:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.834588:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757808.834592:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.834594:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757808.834597:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757808.834604:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787757808.834605:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757808.834628:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757808.834628:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834641:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757808.834650:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757808.834651:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659685248/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757808.834663:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757808.834674:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.834681:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757808.834686:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834687:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305536014 00000100:00000001:2.0:1787757808.834695:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305536705 00000100:00000001:3.0:1787757808.834696:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.834711:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305536014 00000100:00080000:3.0:1787757808.834713:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757808.834724:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305536705 00000100:00080000:3.0:1787757808.834726:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757808.834740:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757808.834741:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757808.834755:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757808.834757:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757808.834771:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757808.834773:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757808.834785:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10863 00000100:00100000:2.0:1787757808.834787:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757808/real 1787757808] req@ffff8ffb45c30e00 x1874588659685120/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757808.834797:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757808.834811:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:2.0:1787757808.834815:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757808.834817:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:2.0:1787757808.834820:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000020:00000001:3.0:1787757808.834826:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757808.834830:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000020:00000001:3.0:1787757808.834830:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757808.834838:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757808.834840:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000020:00000001:3.0:1787757808.834854:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757808.834855:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.834872:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757808.834872:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757808.834889:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757808.834890:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757808.834924:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1787757808.834925:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659685120/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757808.834968:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757808.834980:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757808.834990:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757808.834993:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757808.834996:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835000:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757808.835002:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757808.835009:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757808.835012:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787757808.835017:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835020:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757808.835026:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835028:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835041:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:3.0:1787757808.835045:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757808.835050:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.835054:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757808.835057:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757808.835061:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.835073:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835086:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:3.0:1787757808.835088:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659685376/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757808.835099:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757808.835124:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757808.835133:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757808.835143:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757808.835144:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757808.835172:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757808.835175:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685120:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757808.835234:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835239:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757808.835242:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757808.835245:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757808.835253:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757808.835256:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757808.835259:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757808.835263:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757808.835265:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757808.835269:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757808.835271:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835273:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835278:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757808.835280:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835283:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757808.835285:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757808.835287:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757808.835290:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757808.835293:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757808.835297:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757808.835301:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757808.835306:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757808.835312:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685248:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757808.835317:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757808.835321:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757808.835324:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757808.835327:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757808.835333:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835344:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757808.835347:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:2.0:1787757808.835356:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757808.835387:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757808.835398:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757808.835402:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757808.835405:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835409:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757808.835414:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835422:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305536012 00000100:00000001:2.0:1787757808.835428:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305536705 00000100:00000001:2.0:1787757808.835433:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305536012 00000100:00000001:2.0:1787757808.835438:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305536705 00000100:00000001:2.0:1787757808.835443:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757808.835447:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835451:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757808.835461:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757808/real 1787757808] req@ffff8ffb45c31c00 x1874588659685248/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757808.835473:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757808.835475:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757808.835479:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757808.835482:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757808.835484:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835486:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757808.835488:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757808.835493:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659685248/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757808.835503:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835505:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757808.835509:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835511:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757808.835513:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835515:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757808.835534:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757808.835537:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757808.835541:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757808.835543:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757808.835548:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685248:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757808.835556:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757808.835559:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757808.835561:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757808.835565:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757808.835567:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757808.835570:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757808.835574:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685376:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757808.835578:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757808.835581:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757808.835583:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757808.835586:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757808.835590:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835597:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757808.835599:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:2.0:1787757808.835605:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757808.835622:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757808.835627:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757808.835630:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757808.835632:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835634:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757808.835638:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835642:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305536012 00000100:00000001:2.0:1787757808.835645:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305536705 00000100:00000001:2.0:1787757808.835648:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305536012 00000100:00000001:2.0:1787757808.835652:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305536705 00000100:00000001:2.0:1787757808.835654:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757808.835657:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835659:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757808.835664:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757808/real 1787757808] req@ffff8ffb45c33b80 x1874588659685376/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757808.835673:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757808.835675:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757808.835678:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757808.835680:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757808.835682:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835684:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757808.835686:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757808.835691:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659685376/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757808.835699:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835701:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757808.835704:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835706:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757808.835707:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835709:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757808.835728:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757808.835731:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757808.835734:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757808.835737:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757808.835740:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685376:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757808.835744:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835747:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757808.835748:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757808.835750:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757808.835753:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757808.835755:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757808.835757:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757808.835760:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757808.835761:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757808.835764:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757808.835766:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835768:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757808.835770:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757808.835772:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757808.835775:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757808.835778:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757808.835779:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757808.835782:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757808.835783:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757808.835786:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757808.835787:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835790:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757808.835794:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757808.835796:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757808.835798:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757808.835800:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757809.409333:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757809.409342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757809.409345:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757809.409352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757809.409356:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757809.409361:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757809.409364:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757809.409366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757809.409370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757809.601329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757809.601339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757809.601351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757809.601355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757809.601435:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757809.601441:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757809.601446:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757810.433327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757810.433336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757810.433339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757810.433345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757810.433350:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757810.433355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757810.433358:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757810.433362:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757810.433366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757810.625317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757810.625326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757810.625335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757810.625337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757810.625363:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757810.625367:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757810.625370:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757811.457282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757811.457292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757811.457295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757811.457302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757811.457307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757811.457313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757811.457318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757811.457324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757811.457329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757811.649312:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757811.649321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757811.649330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757811.649332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757811.649445:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757811.649455:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757811.649462:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757812.481348:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757812.481356:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757812.481359:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757812.481366:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757812.481370:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757812.481375:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757812.481379:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757812.481382:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757812.481386:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757812.673298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757812.673307:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757812.673316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757812.673321:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757812.673324:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757812.673329:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757812.673333:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757812.673336:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757812.673339:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757812.673343:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757812.673372:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757812.673376:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757812.673379:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757813.505293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757813.505303:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757813.505305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757813.505312:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757813.505316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757813.505321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757813.505325:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757813.505328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757813.505332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757813.697306:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757813.697315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757813.697323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757813.697325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757813.697408:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757813.697419:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757813.697425:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757813.953422:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757813.953439:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757813.953443:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757813.953449:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757813.953457:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757813.953468:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10868 00000100:00080000:2.0:1787757813.953486:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757813.953492:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757813.953495:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757813.953502:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757813.953507:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757813.953514:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757813.953519:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757813.953524:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757813.953527:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757813.953536:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757813.953542:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757813.953547:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757813.953554:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757813.953560:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.953566:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757813.953571:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757813.953595:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757813.953599:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.953606:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757813.953612:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757813.953621:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.953645:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757813.953679:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659685504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757813.953695:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757813.953702:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.953729:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757813.953734:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757813.953739:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757813.953741:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757813.953743:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757813.953747:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757813.953751:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10868 00000100:00080000:2.0:1787757813.953756:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757813.953759:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757813.953761:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757813.953773:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757813.953777:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757813.953782:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757813.953786:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757813.953791:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757813.953794:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757813.953800:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757813.953804:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757813.953806:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757813.953819:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757813.953821:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757813.953824:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.953837:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757813.953864:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757813.953873:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:2.0:1787757813.953875:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757813.953883:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:2.0:1787757813.953883:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.953899:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.953901:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757813.953925:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757813.953927:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757813.953939:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.953942:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757813.953946:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757813.953955:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757813.953957:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757813.953973:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757813.953980:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659685504:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757813.953991:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.953993:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757813.954003:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757813.954009:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.954017:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:2.0:1787757813.954020:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659685632/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757813.954029:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.954054:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757813.954060:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757813.954061:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954067:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:0.0:1787757813.954076:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757813.954079:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757813.954087:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757813.954093:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757813.954096:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757813.954099:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757813.954104:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757813.954109:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10868 00000100:00080000:2.0:1787757813.954120:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757813.954124:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757813.954127:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757813.954134:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000800:00000100:0.0:1787757813.954142:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787757813.954143:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757813.954155:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757813.954162:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757813.954166:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757813.954167:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757813.954170:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757813.954177:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757813.954235:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954239:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757813.954249:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954255:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305541826 00000100:00080000:2.0:1787757813.954265:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757813.954270:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757813.954272:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757813.954276:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757813.954282:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757813.954288:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954293:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305541826 00000100:00000001:2.0:1787757813.954293:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757813.954306:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757813.954311:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00000001:2.0:1787757813.954314:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.954320:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954321:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757813.954331:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757813.954332:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757813.954337:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757813.954337:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.954349:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757813.954353:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757813.954359:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757813.954361:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757813/real 1787757813] req@ffff8ffb47a38380 x1874588659685504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757813.954377:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757813.954394:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00100000:2.0:1787757813.954397:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659685760/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787757813.954399:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757813.954409:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757813.954421:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757813.954423:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757813.954427:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757813.954429:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954434:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757813.954437:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954438:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757813.954459:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757813.954469:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757813.954472:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00100000:0.0:1787757813.954473:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659685504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757813.954475:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757813.954485:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757813.954495:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10868 00000100:00080000:2.0:1787757813.954505:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757813.954515:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757813.954522:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757813.954530:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757813.954533:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757813.954543:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757813.954546:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:2.0:1787757813.954552:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757813.954559:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757813.954563:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757813.954570:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757813.954575:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757813.954576:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.954586:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757813.954586:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757813.954593:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757813.954602:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757813.954607:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1787757813.954614:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757813.954621:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954623:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:2.0:1787757813.954630:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757813.954639:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787757813.954641:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659685504:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757813.954659:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954664:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757813.954666:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757813.954676:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00100000:2.0:1787757813.954677:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757813.954687:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000008:00000001:2.0:1787757813.954687:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.954694:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:2.0:1787757813.954696:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.954712:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:2.0:1787757813.954713:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757813.954720:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757813.954721:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:2.0:1787757813.954736:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000020:00000001:0.0:1787757813.954737:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757813.954753:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757813.954757:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757813.954759:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.954765:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757813.954773:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757813.954774:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954790:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.954800:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757813.954802:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659685888/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757813.954813:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757813.954816:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757813.954819:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757813.954825:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954830:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757813.954834:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954837:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757813.954868:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757813.954871:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757813.954874:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.954879:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757813.954882:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757813.954887:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757813.954898:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685632:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757813.954905:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757813.954908:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757813.954911:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.954913:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757813.954919:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954928:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757813.954930:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:0.0:1787757813.954936:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757813.954959:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757813.954966:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757813.954970:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757813.954972:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954975:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757813.954979:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.954984:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305541134 00000100:00000001:0.0:1787757813.954988:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305541826 00000100:00000001:0.0:1787757813.954991:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305541134 00000100:00000001:0.0:1787757813.954994:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305541826 00000100:00000001:0.0:1787757813.954997:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757813.954999:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955001:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757813.955007:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757813/real 1787757813] req@ffff8ffb47a38000 x1874588659685632/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757813.955018:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757813.955019:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757813.955022:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757813.955024:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757813.955026:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955029:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757813.955031:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757813.955036:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659685632/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757813.955044:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955047:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757813.955051:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955053:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757813.955055:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955058:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757813.955075:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757813.955079:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757813.955083:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757813.955086:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757813.955090:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685632:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757813.955095:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757813.955097:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757813.955099:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.955102:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757813.955104:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757813.955106:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757813.955110:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685760:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757813.955114:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757813.955117:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757813.955119:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.955121:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757813.955124:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955130:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757813.955132:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757813.955136:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757813.955148:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757813.955152:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757813.955155:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757813.955157:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955159:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757813.955162:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955166:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305541131 00000100:00000001:0.0:1787757813.955169:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305541826 00000100:00000001:0.0:1787757813.955172:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305541131 00000100:00000001:0.0:1787757813.955175:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305541826 00000100:00000001:0.0:1787757813.955178:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757813.955181:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955183:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757813.955222:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757813/real 1787757813] req@ffff8ffb47a3b480 x1874588659685760/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757813.955233:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757813.955235:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757813.955238:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757813.955240:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757813.955242:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955244:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757813.955246:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757813.955250:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659685760/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757813.955259:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955262:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757813.955265:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955267:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757813.955268:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955270:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757813.955285:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757813.955289:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757813.955293:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757813.955296:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757813.955300:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685760:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757813.955304:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757813.955307:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757813.955308:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.955311:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757813.955313:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757813.955315:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757813.955320:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685888:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757813.955324:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757813.955326:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757813.955329:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757813.955331:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757813.955335:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955340:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757813.955342:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757813.955347:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757813.955360:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757813.955365:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757813.955368:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757813.955370:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955372:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757813.955376:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955380:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305541132 00000100:00000001:0.0:1787757813.955383:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305541826 00000100:00000001:0.0:1787757813.955386:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305541132 00000100:00000001:0.0:1787757813.955389:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305541826 00000100:00000001:0.0:1787757813.955392:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757813.955394:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955396:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757813.955401:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757813/real 1787757813] req@ffff8ffb47a39180 x1874588659685888/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757813.955409:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757813.955411:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757813.955414:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757813.955415:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757813.955417:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955419:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757813.955422:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757813.955426:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659685888/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757813.955443:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955445:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757813.955449:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955450:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757813.955452:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955454:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757813.955474:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757813.955479:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757813.955485:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757813.955488:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757813.955494:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659685888:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757813.955502:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955506:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757813.955508:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757813.955511:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757813.955516:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757813.955519:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757813.955522:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757813.955526:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757813.955529:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757813.955533:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757813.955536:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757813.955542:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757813.955544:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757813.955549:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757813.955551:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757813.955553:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757813.955557:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757813.955560:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757813.955563:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757813.955566:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955570:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757813.955572:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757813.955574:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757813.955577:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757813.955579:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757813.955580:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757813.955582:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757813.955584:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757813.955587:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757813.955589:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955591:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757813.955596:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757813.955598:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757813.955600:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757813.955603:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757814.529297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757814.529310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757814.529314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757814.529323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757814.529330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757814.529337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757814.529342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757814.529347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757814.529352:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757814.721398:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757814.721409:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757814.721424:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757814.721426:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757814.721461:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757814.721464:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757814.721469:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757815.553279:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757815.553292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757815.553296:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757815.553306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757815.553312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757815.553318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757815.553323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757815.553329:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757815.553334:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757815.745307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757815.745317:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757815.745321:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757815.745325:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757815.745332:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757815.745335:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757815.745339:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757815.745342:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757815.745347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757815.745349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757815.745500:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757815.745510:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757815.745517:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757816.577282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757816.577291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757816.577294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757816.577301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757816.577306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757816.577310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757816.577314:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757816.577317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757816.577321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757816.769323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757816.769330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757816.769338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757816.769340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757816.769368:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757816.769371:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757816.769375:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757817.601278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757817.601287:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757817.601290:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757817.601296:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757817.601300:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757817.601305:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757817.601309:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757817.601312:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757817.601316:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757817.793356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757817.793363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757817.793369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757817.793371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757817.793454:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757817.793460:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757817.793464:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757818.625332:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757818.625341:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757818.625344:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757818.625350:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757818.625355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757818.625360:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757818.625363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757818.625366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757818.625370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757818.817313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757818.817323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757818.817334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757818.817338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757818.817375:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757818.817379:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757818.817382:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757819.073366:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757819.073379:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757819.073382:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757819.073386:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757819.073392:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757819.073401:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10873 00000100:00080000:3.0:1787757819.073415:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757819.073419:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757819.073422:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757819.073427:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757819.073430:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757819.073434:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757819.073437:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757819.073440:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757819.073442:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757819.073448:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757819.073452:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757819.073455:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757819.073460:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757819.073464:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073468:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757819.073472:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757819.073488:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757819.073491:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073496:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757819.073500:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757819.073507:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073527:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757819.073545:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659686016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757819.073567:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757819.073575:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073603:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757819.073609:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757819.073613:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757819.073615:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757819.073618:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757819.073622:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757819.073625:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10873 00000100:00080000:3.0:1787757819.073631:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757819.073634:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757819.073636:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757819.073639:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757819.073641:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757819.073644:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757819.073647:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757819.073649:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757819.073651:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757819.073655:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757819.073659:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757819.073661:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757819.073667:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757819.073670:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073672:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757819.073678:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757819.073682:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073685:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757819.073691:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757819.073693:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073697:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757819.073699:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757819.073702:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073710:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757819.073718:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659686144/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757819.073729:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757819.073732:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.073736:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757819.073737:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757819.073746:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757819.073775:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757819.073779:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757819.073780:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757819.073789:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757819.073790:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757819.073797:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757819.073799:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757819.073807:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757819.073810:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10873 02000000:00000001:0.0:1787757819.073820:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787757819.073824:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787757819.073829:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757819.073833:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757819.073841:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757819.073843:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757819.073854:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:0.0:1787757819.073864:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659686016:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787757819.073864:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757819.073874:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757819.073878:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:3.0:1787757819.073882:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757819.073887:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757819.073889:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757819.073895:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073896:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757819.073905:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757819.073912:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787757819.073917:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073918:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757819.073929:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757819.073935:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757819.073939:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073942:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757819.073946:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757819.073947:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757819.073955:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000008:00000001:3.0:1787757819.073956:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073965:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.073966:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787757819.073975:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757819.073980:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073984:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757819.073986:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757819.073989:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.073999:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757819.074007:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659686272/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757819.074019:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757819.074024:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757819.074028:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757819.074033:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757819.074040:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757819.074047:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757819.074047:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757819.074054:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757819.074058:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757819.074060:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757819.074064:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757819.074066:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074075:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757819.074076:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10873 00000100:00000001:0.0:1787757819.074088:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757819.074091:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757819.074096:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305546945 00000100:00000001:3.0:1787757819.074097:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757819.074103:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757819.074109:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757819.074111:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757819.074113:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757819.074116:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787757819.074117:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757819.074126:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757819.074128:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757819.074131:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305546945 00000100:00080000:3.0:1787757819.074135:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757819.074142:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074144:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757819.074146:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757819.074154:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757819.074156:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757819.074162:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074164:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.074168:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757819.074172:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:3.0:1787757819.074174:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757819.074181:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757819.074252:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757819/real 1787757819] req@ffff8ffb45c30a80 x1874588659686016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757819.074270:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757819.074273:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757819.074278:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757819.074283:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757819.074287:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757819.074292:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074294:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757819.074298:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757819.074305:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659686016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757819.074316:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757819.074320:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757819.074324:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074327:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757819.074336:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757819.074337:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757819.074346:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757819.074347:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757819.074355:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757819.074356:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074359:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757819.074366:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757819.074371:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757819.074372:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757819.074386:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1787757819.074393:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659686400/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757819.074394:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757819.074409:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659686016:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757819.074425:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757819.074431:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757819.074434:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074461:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757819.074465:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757819.074467:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757819.074471:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757819.074474:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757819.074477:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757819.074483:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686144:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757819.074488:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757819.074491:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757819.074494:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757819.074497:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757819.074503:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074510:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757819.074512:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787757819.074518:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757819.074535:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757819.074541:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757819.074544:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757819.074546:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074549:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757819.074552:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074557:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305546254 00000100:00000001:0.0:1787757819.074561:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305546945 00000100:00000001:0.0:1787757819.074564:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305546254 00000100:00000001:0.0:1787757819.074567:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305546945 00000100:00000001:0.0:1787757819.074570:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757819.074572:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074575:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757819.074580:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757819/real 1787757819] req@ffff8ffb45c31500 x1874588659686144/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757819.074590:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757819.074592:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757819.074595:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757819.074597:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757819.074599:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074601:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757819.074603:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757819.074608:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659686144/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757819.074625:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074628:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757819.074631:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074633:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757819.074635:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074637:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757819.074651:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757819.074654:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757819.074658:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757819.074661:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757819.074664:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686144:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757819.074670:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074672:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757819.074675:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757819.074678:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757819.074683:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757819.074686:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757819.074688:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757819.074692:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757819.074695:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757819.074698:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757819.074700:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074702:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757819.074704:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757819.074705:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757819.074707:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757819.074709:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757819.074710:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757819.074712:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757819.074714:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757819.074715:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757819.074717:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074720:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074728:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757819.074731:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074734:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757819.074736:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757819.074737:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757819.074739:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757819.074741:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757819.074744:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757819.074746:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757819.074748:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757819.074778:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686272:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757819.074783:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757819.074785:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757819.074787:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757819.074790:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757819.074793:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074800:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757819.074802:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757819.074807:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757819.074821:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757819.074827:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757819.074830:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757819.074831:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074834:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757819.074837:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074841:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305546251 00000100:00000001:0.0:1787757819.074844:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305546945 00000100:00000001:0.0:1787757819.074847:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305546251 00000100:00000001:0.0:1787757819.074850:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305546945 00000100:00000001:0.0:1787757819.074853:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757819.074855:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074858:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757819.074863:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757819/real 1787757819] req@ffff8ffb45c30700 x1874588659686272/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757819.074880:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757819.074882:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757819.074885:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757819.074887:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757819.074888:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074891:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757819.074893:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757819.074898:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659686272/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757819.074906:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074909:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757819.074911:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074913:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757819.074915:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.074917:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757819.074929:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757819.074932:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757819.074935:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757819.074938:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757819.074942:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686272:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757819.074947:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757819.074948:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757819.074951:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757819.074954:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757819.074955:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757819.074958:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757819.074961:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686400:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757819.074965:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757819.074967:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757819.074970:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757819.074972:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757819.074975:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.074980:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757819.074982:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757819.074986:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757819.074997:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757819.075002:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757819.075004:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757819.075006:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.075008:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757819.075011:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.075015:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305546251 00000100:00000001:0.0:1787757819.075018:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305546945 00000100:00000001:0.0:1787757819.075021:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305546251 00000100:00000001:0.0:1787757819.075024:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305546945 00000100:00000001:0.0:1787757819.075027:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757819.075029:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.075031:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757819.075036:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757819/real 1787757819] req@ffff8ffb45c32a00 x1874588659686400/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757819.075044:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757819.075046:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757819.075048:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757819.075050:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757819.075052:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.075054:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757819.075056:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757819.075060:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659686400/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757819.075068:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.075071:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757819.075074:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.075076:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757819.075078:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.075080:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757819.075093:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757819.075096:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757819.075100:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757819.075103:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757819.075106:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686400:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757819.075110:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.075112:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757819.075114:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757819.075115:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757819.075117:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757819.075119:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757819.075120:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757819.075123:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757819.075124:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757819.075126:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757819.075128:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.075130:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757819.075132:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757819.075133:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757819.075135:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757819.075137:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757819.075138:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757819.075140:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757819.075141:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757819.075143:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757819.075145:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.075147:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757819.075150:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757819.075152:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757819.075154:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757819.075156:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757819.649321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757819.649330:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757819.649332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757819.649340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757819.649344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757819.649349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757819.649353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757819.649356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757819.649360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757819.841360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757819.841367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757819.841377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757819.841380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757819.841461:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757819.841468:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757819.841474:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757820.673292:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757820.673301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757820.673303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757820.673309:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757820.673314:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757820.673318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757820.673321:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757820.673324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757820.673328:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757820.865311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757820.865319:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757820.865329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757820.865331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757820.865369:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757820.865373:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757820.865376:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757821.697348:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757821.697357:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757821.697360:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757821.697366:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757821.697371:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757821.697375:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757821.697378:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757821.697382:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757821.697385:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757821.889298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757821.889306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757821.889314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757821.889316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757821.889418:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757821.889423:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757821.889427:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757822.721310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757822.721319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757822.721322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757822.721329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757822.721333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757822.721338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757822.721341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757822.721345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757822.721348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757822.913309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757822.913317:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757822.913325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757822.913331:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757822.913335:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757822.913340:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757822.913344:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757822.913347:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757822.913350:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757822.913353:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757822.913383:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757822.913386:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757822.913389:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757823.745321:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757823.745331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757823.745333:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757823.745339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757823.745345:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757823.745350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757823.745353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757823.745356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757823.745360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757823.937361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757823.937373:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757823.937384:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757823.937387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757823.937510:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757823.937518:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757823.937524:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757824.193337:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757824.193350:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757824.193353:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757824.193358:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757824.193364:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757824.193373:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10878 00000100:00080000:1.0:1787757824.193387:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757824.193391:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757824.193393:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757824.193399:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757824.193402:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757824.193407:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757824.193410:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757824.193413:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757824.193415:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757824.193421:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757824.193424:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757824.193427:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757824.193432:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757824.193435:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193438:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757824.193441:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757824.193456:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757824.193458:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193462:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757824.193466:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757824.193471:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193486:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757824.193498:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659686528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757824.193512:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757824.193518:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193540:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757824.193544:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757824.193549:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757824.193551:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757824.193553:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757824.193556:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757824.193559:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10878 00000100:00080000:1.0:1787757824.193563:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757824.193566:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757824.193568:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757824.193570:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757824.193572:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757824.193575:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757824.193579:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757824.193581:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757824.193583:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757824.193588:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757824.193592:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757824.193595:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757824.193601:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757824.193604:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193607:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757824.193614:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757824.193618:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193622:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.193623:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787757824.193628:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757824.193634:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757824.193643:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787757824.193649:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193655:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757824.193657:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757824.193679:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193686:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.193687:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757824.193698:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757824.193699:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659686656/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757824.193713:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757824.193721:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757824.193722:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757824.193728:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757824.193731:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193736:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757824.193742:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757824.193747:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659686528:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787757824.193749:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757824.193757:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757824.193764:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757824.193765:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757824.193773:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757824.193774:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757824.193782:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757824.193787:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10878 02000000:00000001:0.0:1787757824.193789:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:1.0:1787757824.193799:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757824.193802:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757824.193803:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193805:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757824.193813:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757824.193815:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757824.193817:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757824.193821:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757824.193824:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757824.193826:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757824.193829:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787757824.193831:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757824.193836:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:1.0:1787757824.193839:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757824.193845:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757824.193847:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787757824.193852:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757824.193857:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193860:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757824.193863:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757824.193868:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193871:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757824.193880:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757824.193883:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193886:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757824.193888:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757824.193891:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193899:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757824.193907:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1787757824.193914:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659686784/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757824.193928:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757824.193930:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757824.193932:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193940:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.193941:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787757824.193945:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757824.193948:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757824.193951:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757824.193954:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757824.193957:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757824.193963:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757824.193967:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.193968:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757824.193977:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10878 00000100:00000001:0.0:1787757824.193979:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305552065 00000100:00080000:1.0:1787757824.193983:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757824.193988:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757824.193991:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757824.193994:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757824.193995:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:1.0:1787757824.193997:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757824.194005:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757824.194009:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757824.194010:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305552065 00000100:00000001:1.0:1787757824.194013:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757824.194019:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757824.194024:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787757824.194029:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757824.194036:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757824.194038:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757824.194042:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757824.194044:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194048:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757824.194049:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757824.194056:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757824.194058:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787757824.194062:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194066:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757824.194069:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757824.194082:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757824/real 1787757824] req@ffff8ffb4a537b80 x1874588659686528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787757824.194091:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757824.194097:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.194104:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000400:00000001:0.0:1787757824.194105:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:1.0:1787757824.194107:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000400:00000001:0.0:1787757824.194112:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:1.0:1787757824.194114:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194122:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757824.194127:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194130:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757824.194136:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757824.194143:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659686912/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757824.194144:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757824.194150:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757824.194165:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:0.0:1787757824.194167:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659686528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757824.194173:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.194182:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194256:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194262:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757824.194266:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194269:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757824.194271:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194274:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757824.194277:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757824.194281:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757824.194287:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757824.194291:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757824.194298:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659686528:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757824.194305:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194308:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757824.194310:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757824.194315:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757824.194321:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757824.194323:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757824.194326:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757824.194331:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757824.194333:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757824.194336:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757824.194338:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194341:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194349:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757824.194352:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194355:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757824.194357:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757824.194359:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757824.194361:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757824.194363:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757824.194367:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757824.194371:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757824.194375:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757824.194382:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686656:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757824.194389:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757824.194392:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757824.194395:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757824.194399:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757824.194404:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194415:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757824.194418:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:0.0:1787757824.194425:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757824.194453:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757824.194463:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757824.194467:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757824.194471:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194474:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757824.194480:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194488:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305551374 00000100:00000001:0.0:1787757824.194492:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305552065 00000100:00000001:0.0:1787757824.194496:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305551374 00000100:00000001:0.0:1787757824.194499:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305552065 00000100:00000001:0.0:1787757824.194502:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757824.194504:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194507:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757824.194513:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757824/real 1787757824] req@ffff8ffb4a535c00 x1874588659686656/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757824.194525:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757824.194526:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757824.194530:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757824.194532:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757824.194534:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194536:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757824.194539:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757824.194543:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659686656/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757824.194561:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194564:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757824.194568:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194570:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757824.194571:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194573:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757824.194588:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757824.194591:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757824.194596:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757824.194600:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757824.194607:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686656:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757824.194616:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757824.194618:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757824.194621:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757824.194625:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757824.194629:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757824.194633:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757824.194639:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686784:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757824.194646:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757824.194650:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757824.194653:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757824.194657:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757824.194684:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194698:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757824.194701:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787757824.194708:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757824.194731:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757824.194738:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757824.194742:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757824.194744:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194747:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757824.194750:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194755:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305551371 00000100:00000001:0.0:1787757824.194758:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305552065 00000100:00000001:0.0:1787757824.194761:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305551371 00000100:00000001:0.0:1787757824.194764:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305552065 00000100:00000001:0.0:1787757824.194766:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757824.194769:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194771:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757824.194777:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757824/real 1787757824] req@ffff8ffb4a534700 x1874588659686784/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757824.194786:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757824.194788:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757824.194791:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757824.194793:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757824.194795:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194797:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757824.194799:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757824.194804:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659686784/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757824.194812:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194816:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757824.194818:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194820:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757824.194822:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194824:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757824.194837:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757824.194840:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757824.194844:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757824.194846:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757824.194850:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686784:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757824.194855:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757824.194857:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757824.194859:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757824.194862:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757824.194864:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757824.194867:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757824.194870:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686912:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757824.194874:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757824.194876:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757824.194878:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757824.194881:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757824.194885:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194890:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757824.194892:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:0.0:1787757824.194897:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757824.194909:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757824.194914:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757824.194917:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757824.194919:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194921:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757824.194924:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194928:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305551371 00000100:00000001:0.0:1787757824.194931:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305552065 00000100:00000001:0.0:1787757824.194934:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305551371 00000100:00000001:0.0:1787757824.194937:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305552065 00000100:00000001:0.0:1787757824.194940:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757824.194942:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.194944:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757824.194949:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757824/real 1787757824] req@ffff8ffb4a534a80 x1874588659686912/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757824.194966:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757824.194967:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757824.194970:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757824.194972:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757824.194974:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194976:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757824.194977:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757824.194982:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659686912/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757824.194990:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194993:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757824.194996:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.194998:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757824.194999:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.195002:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757824.195014:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757824.195016:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757824.195020:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757824.195022:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757824.195026:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659686912:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757824.195031:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.195033:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757824.195035:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757824.195037:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757824.195040:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757824.195042:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757824.195044:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757824.195046:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757824.195048:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757824.195050:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757824.195052:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.195055:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757824.195056:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757824.195058:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757824.195060:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757824.195062:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757824.195063:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757824.195066:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757824.195068:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757824.195070:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757824.195071:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.195073:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757824.195074:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757824.195076:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757824.195077:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757824.195078:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757824.195080:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757824.195082:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757824.195084:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757824.195085:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757824.195087:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.195089:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757824.195094:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757824.195095:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757824.195098:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757824.195100:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757824.769275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757824.769284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757824.769287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757824.769293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757824.769298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757824.769302:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757824.769305:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757824.769308:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757824.769312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757824.961314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757824.961323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757824.961334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757824.961337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757824.961376:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757824.961380:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757824.961384:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757825.793286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757825.793295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757825.793298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757825.793304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757825.793309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757825.793314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757825.793317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757825.793320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757825.793324:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757825.985321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757825.985333:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757825.985337:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757825.985340:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757825.985347:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757825.985351:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757825.985354:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757825.985358:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757825.985363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757825.985365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757825.985451:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757825.985457:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757825.985462:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757826.817311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757826.817320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757826.817323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757826.817330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757826.817335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757826.817340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757826.817343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757826.817346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757826.817350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757827.009320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757827.009328:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757827.009336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757827.009338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757827.009365:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757827.009369:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757827.009372:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757827.841373:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757827.841385:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757827.841388:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757827.841398:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757827.841405:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757827.841411:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757827.841417:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757827.841422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757827.841427:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757828.033350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757828.033362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757828.033373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757828.033375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757828.033516:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757828.033524:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757828.033531:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757828.865342:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757828.865357:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757828.865360:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757828.865368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757828.865374:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757828.865380:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757828.865384:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757828.865389:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757828.865393:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757829.057335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757829.057346:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757829.057358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757829.057363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757829.057403:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757829.057408:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757829.057414:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757829.313411:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757829.313423:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757829.313426:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757829.313431:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757829.313437:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757829.313446:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10884 00000100:00080000:2.0:1787757829.313460:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757829.313464:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757829.313466:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757829.313472:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757829.313475:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757829.313479:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757829.313482:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757829.313485:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757829.313487:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757829.313492:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757829.313496:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757829.313499:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757829.313504:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757829.313508:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.313510:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757829.313513:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757829.313525:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757829.313527:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.313531:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757829.313534:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757829.313539:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.313553:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757829.313565:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659687040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757829.313613:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757829.313619:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.313645:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757829.313650:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757829.313654:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757829.313657:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757829.313659:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757829.313663:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757829.313666:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10884 00000100:00080000:2.0:1787757829.313672:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757829.313675:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757829.313677:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757829.313681:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757829.313683:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757829.313686:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757829.313690:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757829.313693:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757829.313695:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757829.313701:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757829.313705:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757829.313708:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757829.313716:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757829.313716:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757829.313725:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.313726:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757829.313728:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757829.313733:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:2.0:1787757829.313738:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757829.313739:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757829.313746:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757829.313748:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.313755:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.313757:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757829.313763:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757829.313764:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757829.313770:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.313772:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.313779:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757829.313784:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:0.0:1787757829.313787:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659687040:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757829.313789:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.313802:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757829.313805:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.313810:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757829.313816:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757829.313819:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659687168/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757829.313821:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757829.313832:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.313837:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757829.313842:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.313848:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757829.313852:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757829.313857:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757829.313858:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757829.313860:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757829.313864:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00080000:2.0:1787757829.313868:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757829.313872:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757829.313883:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757829.313890:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10884 00000100:00080000:2.0:1787757829.313899:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757829.313903:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757829.313907:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757829.313912:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757829.313916:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757829.313919:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000800:00000100:0.0:1787757829.313928:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787757829.313928:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757829.313941:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757829.313947:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757829.313950:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757829.313959:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:2.0:1787757829.313960:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757829.313964:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.313973:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757829.313975:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.313982:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757829.313983:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757829.313992:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757829.313993:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305557185 02000000:00000001:2.0:1787757829.314003:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.314008:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757829.314013:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00100000:2.0:1787757829.314013:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757829.314025:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314027:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305557185 00000100:00000001:2.0:1787757829.314029:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314043:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757829.314045:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757829.314051:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.314054:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757829.314056:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757829.314059:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314067:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757829.314071:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314078:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314087:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757829.314091:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659687296/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757829.314106:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757829/real 1787757829] req@ffff8ffb47a3a300 x1874588659687040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757829.314122:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757829.314133:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757829.314149:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757829.314149:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757829.314156:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:2.0:1787757829.314160:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757829.314168:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:2.0:1787757829.314170:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757829.314176:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757829.314179:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757829.314182:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314234:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757829.314238:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757829.314250:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659687040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757829.314255:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757829.314270:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757829.314283:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757829.314284:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10884 00000100:00000001:0.0:1787757829.314296:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757829.314301:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757829.314310:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757829.314312:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757829.314320:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757829.314322:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757829.314328:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757829.314332:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757829.314337:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000020:00000001:2.0:1787757829.314340:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757829.314344:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:2.0:1787757829.314348:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:0.0:1787757829.314352:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000020:00000001:2.0:1787757829.314362:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:0.0:1787757829.314367:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757829.314370:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757829.314378:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757829.314379:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787757829.314394:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659687040:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787757829.314397:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757829.314410:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314412:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757829.314415:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757829.314418:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787757829.314424:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757829.314425:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757829.314433:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757829.314434:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.314440:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757829.314442:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000008:00100000:2.0:1787757829.314444:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000020:00000001:0.0:1787757829.314448:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000008:00000001:2.0:1787757829.314453:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757829.314454:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:2.0:1787757829.314462:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.314465:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757829.314470:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 02000000:00000001:2.0:1787757829.314472:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757829.314475:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 02000000:00000001:2.0:1787757829.314478:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314481:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757829.314484:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757829.314487:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757829.314490:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757829.314497:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314500:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757829.314505:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314507:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314518:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757829.314522:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757829.314528:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:2.0:1787757829.314528:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659687424/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757829.314535:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757829.314540:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.314548:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757829.314554:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757829.314557:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757829.314565:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.314566:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757829.314599:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757829.314608:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687168:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757829.314614:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757829.314618:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757829.314620:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.314622:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757829.314628:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314639:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757829.314641:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:0.0:1787757829.314647:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757829.314668:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757829.314675:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757829.314679:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757829.314682:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314685:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757829.314689:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314694:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305556494 00000100:00000001:0.0:1787757829.314698:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305557185 00000100:00000001:0.0:1787757829.314701:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305556494 00000100:00000001:0.0:1787757829.314704:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305557185 00000100:00000001:0.0:1787757829.314707:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757829.314709:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314712:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757829.314717:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757829/real 1787757829] req@ffff8ffb47a38700 x1874588659687168/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757829.314728:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757829.314730:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757829.314733:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757829.314735:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757829.314737:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314739:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757829.314741:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757829.314746:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659687168/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757829.314755:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314758:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757829.314761:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314763:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757829.314764:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314767:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757829.314782:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757829.314785:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757829.314789:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757829.314791:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757829.314796:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687168:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757829.314801:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757829.314803:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757829.314805:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.314808:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757829.314810:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757829.314812:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757829.314816:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687296:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757829.314820:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757829.314823:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757829.314825:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.314827:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757829.314830:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314836:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757829.314838:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757829.314843:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757829.314855:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757829.314861:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757829.314863:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757829.314865:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314867:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757829.314870:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314874:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305556491 00000100:00000001:0.0:1787757829.314877:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305557185 00000100:00000001:0.0:1787757829.314880:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305556491 00000100:00000001:0.0:1787757829.314883:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305557185 00000100:00000001:0.0:1787757829.314886:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757829.314888:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.314890:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757829.314895:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757829/real 1787757829] req@ffff8ffb47a39180 x1874588659687296/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757829.314912:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757829.314914:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757829.314917:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757829.314919:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757829.314921:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314922:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757829.314924:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757829.314928:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659687296/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757829.314936:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314939:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757829.314941:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314943:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757829.314945:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314947:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757829.314960:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757829.314963:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757829.314967:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757829.314970:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757829.314974:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687296:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757829.314978:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.314981:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757829.314982:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757829.314984:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757829.314987:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757829.314988:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757829.314990:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757829.314993:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757829.314994:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757829.314998:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757829.314999:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315002:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757829.315004:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757829.315005:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757829.315008:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757829.315010:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757829.315011:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757829.315013:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757829.315015:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757829.315018:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757829.315019:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315022:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315026:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757829.315028:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.315032:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757829.315035:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757829.315037:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757829.315039:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757829.315041:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.315044:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757829.315046:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757829.315049:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757829.315058:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687424:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757829.315066:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757829.315069:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757829.315072:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757829.315076:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757829.315082:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.315092:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757829.315094:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757829.315102:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757829.315129:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757829.315142:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757829.315147:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757829.315150:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.315153:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757829.315159:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.315166:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305556491 00000100:00000001:0.0:1787757829.315169:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305557185 00000100:00000001:0.0:1787757829.315173:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305556491 00000100:00000001:0.0:1787757829.315176:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305557185 00000100:00000001:0.0:1787757829.315179:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757829.315181:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.315220:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757829.315226:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757829/real 1787757829] req@ffff8ffb47a3b480 x1874588659687424/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757829.315238:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757829.315240:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757829.315246:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757829.315249:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757829.315252:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315256:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757829.315259:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757829.315266:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659687424/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757829.315280:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315285:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757829.315290:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315293:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757829.315296:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315300:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757829.315321:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757829.315327:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757829.315334:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757829.315338:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757829.315345:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687424:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757829.315350:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315353:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757829.315356:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757829.315358:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757829.315362:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757829.315363:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757829.315365:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757829.315368:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757829.315370:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757829.315373:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757829.315375:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315377:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757829.315382:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757829.315384:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757829.315386:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757829.315388:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757829.889377:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757829.889394:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757829.889399:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757829.889411:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757829.889418:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757829.889426:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757829.889432:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757829.889438:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757829.889444:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757830.081353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757830.081365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757830.081378:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757830.081382:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757830.081464:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757830.081471:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757830.081475:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757830.913306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757830.913318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757830.913323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757830.913331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757830.913337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757830.913343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757830.913347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757830.913351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757830.913355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757831.105354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757831.105363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757831.105371:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757831.105372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757831.105399:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757831.105402:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757831.105406:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757831.937284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757831.937292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757831.937294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757831.937299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757831.937302:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757831.937307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757831.937309:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757831.937312:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757831.937314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757832.129283:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757832.129288:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757832.129293:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757832.129295:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757832.129356:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757832.129360:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757832.129364:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757832.961228:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757832.961234:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757832.961236:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757832.961239:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757832.961242:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757832.961245:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757832.961247:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757832.961249:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757832.961251:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757833.153297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757833.153304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757833.153313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757833.153317:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757833.153321:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757833.153326:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757833.153330:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757833.153333:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757833.153336:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757833.153339:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757833.153368:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757833.153372:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757833.153375:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757833.985306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757833.985318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757833.985321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757833.985329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757833.985335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757833.985341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757833.985345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757833.985349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757833.985353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757834.177377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757834.177386:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757834.177396:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757834.177398:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757834.177547:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757834.177553:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757834.177558:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757834.433422:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757834.433434:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757834.433438:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757834.433442:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757834.433448:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757834.433457:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10889 00000100:00080000:3.0:1787757834.433471:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757834.433476:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757834.433478:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757834.433513:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757834.433516:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757834.433521:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757834.433524:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757834.433527:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757834.433529:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757834.433534:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757834.433537:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757834.433541:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757834.433545:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757834.433549:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.433552:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757834.433555:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757834.433567:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757834.433570:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.433574:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757834.433577:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757834.433582:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.433596:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757834.433608:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659687552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757834.433622:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757834.433628:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.433652:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757834.433657:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757834.433665:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757834.433667:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757834.433670:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757834.433674:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757834.433677:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10889 00000100:00080000:3.0:1787757834.433682:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757834.433684:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757834.433687:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757834.433692:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757834.433694:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757834.433697:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757834.433701:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757834.433704:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757834.433706:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757834.433710:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787757834.433711:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757834.433721:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.433722:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757834.433730:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757834.433731:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757834.433737:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757834.433740:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757834.433741:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757834.433752:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.433754:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757834.433757:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757834.433762:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:3.0:1787757834.433766:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757834.433769:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757834.433776:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.433784:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757834.433786:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659687552:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757834.433791:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757834.433798:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757834.433798:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.433804:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757834.433809:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757834.433810:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757834.433817:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757834.433818:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.433828:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757834.433833:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.433838:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757834.433845:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659687680/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757834.433862:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757834.433869:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:3.0:1787757834.433871:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757834.433877:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757834.433878:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.433888:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757834.433891:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757834.433895:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757834.433897:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757834.433900:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757834.433905:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757834.433908:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10889 00000100:00080000:3.0:1787757834.433914:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757834.433918:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757834.433921:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757834.433923:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757834.433926:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757834.433928:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757834.433931:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757834.433934:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757834.433939:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:0.0:1787757834.433944:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787757834.433949:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757834.433956:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757834.433960:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757834.433964:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757834.433967:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.433970:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757834.433971:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757834.433980:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757834.433984:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.433985:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757834.433987:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.433995:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757834.433997:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757834.434001:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434006:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757834.434007:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757834.434014:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757834.434016:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434024:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.434025:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757834.434038:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659687808/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757834.434040:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305562305 00000100:00000001:3.0:1787757834.434056:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757834.434059:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434062:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757834.434064:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757834.434073:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757834.434076:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305562305 00000100:00080000:3.0:1787757834.434078:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757834.434087:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757834.434089:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757834.434091:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757834.434093:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757834.434104:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10889 00000100:00000001:0.0:1787757834.434105:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757834.434110:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757834.434112:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.434115:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757834.434122:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757834.434122:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757834.434130:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757834.434133:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757834.434139:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:0.0:1787757834.434140:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757834/real 1787757834] req@ffff8ffb45c31180 x1874588659687552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757834.434143:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757834.434153:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757834.434158:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757834.434173:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787757834.434176:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757834.434177:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757834.434229:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757834.434234:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:0.0:1787757834.434236:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757834.434244:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434248:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757834.434249:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757834.434258:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757834.434259:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757834.434265:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787757834.434268:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434271:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757834.434274:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434277:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757834.434284:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757834.434291:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757834.434292:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659687552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757834.434295:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757834.434300:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757834.434305:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.434316:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434318:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434327:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:3.0:1787757834.434331:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659687936/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757834.434333:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434343:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757834.434348:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757834.434358:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757834.434360:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757834.434367:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757834.434368:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757834.434379:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757834.434381:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757834.434393:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757834.434398:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757834.434403:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659687552:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757834.434413:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434416:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757834.434419:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757834.434422:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757834.434427:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757834.434429:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757834.434432:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757834.434437:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757834.434439:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757834.434442:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757834.434445:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434448:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434456:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757834.434460:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434464:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757834.434467:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757834.434470:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757834.434473:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757834.434476:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.434505:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757834.434510:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757834.434515:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757834.434523:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687680:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757834.434530:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757834.434535:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757834.434538:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.434541:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757834.434548:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434560:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757834.434563:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757834.434571:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757834.434597:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757834.434605:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757834.434609:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757834.434611:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434613:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757834.434617:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434623:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305561614 00000100:00000001:0.0:1787757834.434627:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305562305 00000100:00000001:0.0:1787757834.434630:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305561614 00000100:00000001:0.0:1787757834.434633:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305562305 00000100:00000001:0.0:1787757834.434636:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757834.434638:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434641:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757834.434647:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757834/real 1787757834] req@ffff8ffb45c30000 x1874588659687680/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757834.434660:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757834.434661:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757834.434665:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757834.434667:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757834.434669:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434671:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757834.434673:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757834.434678:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659687680/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757834.434687:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434690:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757834.434693:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434695:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757834.434696:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434698:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757834.434713:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757834.434716:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757834.434719:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757834.434722:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757834.434726:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687680:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757834.434730:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757834.434732:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757834.434734:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.434737:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757834.434739:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757834.434742:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757834.434745:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687808:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757834.434749:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757834.434751:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757834.434754:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.434756:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757834.434760:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434767:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757834.434768:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:0.0:1787757834.434773:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757834.434786:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757834.434791:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757834.434794:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757834.434795:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434797:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757834.434800:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434804:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305561611 00000100:00000001:0.0:1787757834.434807:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305562305 00000100:00000001:0.0:1787757834.434810:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305561611 00000100:00000001:0.0:1787757834.434813:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305562305 00000100:00000001:0.0:1787757834.434816:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757834.434818:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434820:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757834.434825:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757834/real 1787757834] req@ffff8ffb45c33480 x1874588659687808/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757834.434834:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757834.434836:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757834.434838:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757834.434840:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757834.434841:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434843:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757834.434845:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757834.434849:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659687808/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757834.434858:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434861:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757834.434864:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434866:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757834.434868:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.434870:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757834.434882:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757834.434886:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757834.434890:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757834.434892:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757834.434896:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687808:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757834.434900:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757834.434902:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757834.434904:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.434906:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757834.434908:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757834.434910:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757834.434913:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757834.434917:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757834.434919:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757834.434921:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757834.434924:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757834.434929:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434938:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757834.434940:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787757834.434947:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757834.434968:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757834.434976:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757834.434980:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757834.434983:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434987:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757834.434992:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.434998:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305561611 00000100:00000001:0.0:1787757834.435004:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305562305 00000100:00000001:0.0:1787757834.435010:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305561611 00000100:00000001:0.0:1787757834.435014:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305562305 00000100:00000001:0.0:1787757834.435019:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757834.435023:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.435026:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757834.435035:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757834/real 1787757834] req@ffff8ffb45c33800 x1874588659687936/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757834.435048:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757834.435050:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757834.435054:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757834.435056:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757834.435058:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435061:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757834.435063:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757834.435068:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659687936/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757834.435077:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435080:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757834.435083:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435085:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757834.435086:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435088:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757834.435101:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757834.435104:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757834.435108:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757834.435110:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757834.435114:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659687936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757834.435119:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435122:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757834.435124:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757834.435126:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757834.435130:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757834.435132:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757834.435134:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757834.435137:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757834.435138:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757834.435141:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757834.435143:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435145:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757834.435147:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757834.435148:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757834.435151:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757834.435153:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757834.435154:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757834.435157:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757834.435158:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757834.435160:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757834.435161:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435163:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757834.435165:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757834.435166:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757834.435168:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757834.435169:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757834.435171:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757834.435172:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757834.435174:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757834.435175:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757834.435177:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435179:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757834.435184:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757834.435216:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757834.435220:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757834.435222:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757835.009304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757835.009312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757835.009315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757835.009321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757835.009327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757835.009331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757835.009335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757835.009338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757835.009341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757835.201376:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757835.201386:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757835.201391:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757835.201395:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757835.201402:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757835.201405:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757835.201408:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757835.201412:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757835.201418:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757835.201421:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757835.201452:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757835.201455:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757835.201459:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757836.033304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757836.033313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757836.033316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757836.033323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757836.033327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757836.033332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757836.033335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757836.033339:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757836.033343:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757836.225351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757836.225362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757836.225373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757836.225376:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757836.225477:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757836.225483:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757836.225488:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757837.057270:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757837.057279:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757837.057282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757837.057288:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757837.057293:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757837.057297:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757837.057300:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757837.057304:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757837.057308:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757837.249311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757837.249316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757837.249320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757837.249321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757837.249336:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757837.249338:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757837.249340:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757838.081305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757838.081314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757838.081317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757838.081323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757838.081328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757838.081332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757838.081335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757838.081338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757838.081341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757838.273363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757838.273371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757838.273378:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757838.273380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757838.273521:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757838.273527:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757838.273532:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757839.105284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757839.105293:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757839.105296:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757839.105303:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757839.105307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757839.105312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757839.105316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757839.105319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757839.105323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757839.297304:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757839.297314:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757839.297322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757839.297326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757839.297353:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757839.297357:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757839.297360:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757839.553304:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757839.553315:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757839.553319:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757839.553323:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757839.553329:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757839.553338:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10894 00000100:00080000:3.0:1787757839.553351:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757839.553355:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757839.553358:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757839.553363:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757839.553366:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757839.553370:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757839.553373:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757839.553376:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757839.553378:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757839.553384:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757839.553388:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757839.553419:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757839.553423:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757839.553428:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553430:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757839.553434:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757839.553448:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757839.553450:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553454:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757839.553458:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757839.553464:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553480:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757839.553493:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659688064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757839.553508:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757839.553513:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553536:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757839.553540:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757839.553545:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757839.553547:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757839.553549:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757839.553553:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757839.553556:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10894 00000100:00080000:3.0:1787757839.553560:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757839.553563:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757839.553565:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757839.553569:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757839.553570:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757839.553573:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757839.553576:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757839.553581:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757839.553583:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757839.553588:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757839.553593:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757839.553596:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757839.553602:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757839.553605:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553608:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757839.553613:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787757839.553614:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757839.553623:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:3.0:1787757839.553624:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.553630:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757839.553632:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.553637:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757839.553641:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757839.553642:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757839.553647:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.553653:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757839.553654:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757839.553661:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757839.553663:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757839.553665:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.553672:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553677:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757839.553687:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659688064:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787757839.553695:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659688192/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757839.553699:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757839.553708:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757839.553714:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757839.553715:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553722:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.553724:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757839.553731:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.553736:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757839.553738:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757839.553747:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757839.553749:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757839.553752:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757839.553756:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757839.553760:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10894 00000100:00000001:0.0:1787757839.553762:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757839.553765:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757839.553768:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:3.0:1787757839.553771:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757839.553779:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757839.553780:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787757839.553788:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757839.553791:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757839.553793:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757839.553797:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757839.553800:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757839.553803:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757839.553807:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757839.553810:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757839.553812:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757839.553817:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757839.553820:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553823:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757839.553826:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757839.553829:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553831:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757839.553837:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757839.553842:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757839.553846:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757839.553847:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757839.553852:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757839.553857:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553863:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757839.553872:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659688320/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757839.553874:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757839.553885:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757839.553887:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757839.553896:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553896:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553901:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.553907:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787757839.553909:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757839.553920:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757839.553922:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.553925:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757839.553933:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757839.553934:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305567425 00000100:00000001:3.0:1787757839.553950:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757839.553956:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787757839.553960:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10894 00000100:00000001:0.0:1787757839.553967:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305567425 00000100:00000001:0.0:1787757839.553974:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757839.553978:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757839.553981:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.553984:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757839.553994:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757839/real 1787757839] req@ffff8ffb45c33800 x1874588659688064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757839.554010:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757839.554013:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757839.554021:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:3.0:1787757839.554026:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757839.554028:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757839.554033:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757839.554037:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757839.554039:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757839.554045:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757839.554047:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787757839.554053:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659688064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787757839.554061:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757839.554068:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757839.554076:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757839.554077:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554086:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787757839.554090:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757839.554092:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554103:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757839.554105:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757839.554106:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554115:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757839.554115:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757839.554120:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757839.554130:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757839.554138:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1787757839.554141:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757839.554148:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554150:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1787757839.554155:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:0.0:1787757839.554162:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659688064:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757839.554165:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757839.554171:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757839.554173:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554177:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757839.554179:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757839.554182:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000008:00100000:3.0:1787757839.554218:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757839.554222:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.554225:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757839.554232:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757839.554235:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.554237:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757839.554240:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757839.554247:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757839.554248:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.554258:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757839.554261:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757839.554264:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757839.554270:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757839.554276:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00100000:3.0:1787757839.554278:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659688448/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757839.554280:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757839.554288:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757839.554294:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554303:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757839.554305:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757839.554312:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757839.554316:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554318:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757839.554323:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554328:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757839.554331:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757839.554334:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757839.554336:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757839.554338:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.554342:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757839.554345:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757839.554349:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757839.554356:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688192:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757839.554363:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757839.554367:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757839.554370:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.554374:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757839.554381:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554432:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757839.554436:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:0.0:1787757839.554445:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757839.554474:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757839.554485:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757839.554490:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757839.554494:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554497:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757839.554502:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554508:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305566734 00000100:00000001:0.0:1787757839.554512:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305567425 00000100:00000001:0.0:1787757839.554516:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305566734 00000100:00000001:0.0:1787757839.554519:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305567425 00000100:00000001:0.0:1787757839.554521:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757839.554524:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554526:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757839.554533:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757839/real 1787757839] req@ffff8ffb45c33480 x1874588659688192/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757839.554544:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757839.554545:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757839.554549:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757839.554551:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757839.554553:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554555:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757839.554557:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757839.554562:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659688192/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757839.554571:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554574:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757839.554576:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554578:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757839.554581:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554583:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757839.554598:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757839.554602:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757839.554606:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757839.554609:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757839.554614:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688192:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757839.554618:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757839.554620:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757839.554622:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.554624:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757839.554627:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757839.554629:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757839.554633:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688320:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757839.554637:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757839.554639:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757839.554641:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.554644:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757839.554648:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554654:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757839.554656:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757839.554661:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757839.554676:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757839.554681:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757839.554684:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757839.554686:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554688:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757839.554691:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554695:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305566731 00000100:00000001:0.0:1787757839.554698:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305567425 00000100:00000001:0.0:1787757839.554701:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305566731 00000100:00000001:0.0:1787757839.554703:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305567425 00000100:00000001:0.0:1787757839.554707:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757839.554709:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554711:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757839.554716:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757839/real 1787757839] req@ffff8ffb45c30000 x1874588659688320/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757839.554726:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757839.554728:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757839.554731:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757839.554733:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757839.554734:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554736:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757839.554738:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757839.554742:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659688320/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757839.554750:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554752:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757839.554755:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554757:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757839.554759:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554760:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757839.554773:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757839.554775:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757839.554779:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757839.554782:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757839.554786:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688320:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757839.554791:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757839.554792:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757839.554794:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.554796:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757839.554798:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757839.554800:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757839.554805:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688448:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757839.554808:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757839.554811:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757839.554813:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757839.554817:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757839.554822:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554831:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757839.554834:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787757839.554840:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757839.554861:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757839.554868:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757839.554873:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757839.554876:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554879:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757839.554884:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554891:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305566731 00000100:00000001:0.0:1787757839.554896:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305567425 00000100:00000001:0.0:1787757839.554901:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305566731 00000100:00000001:0.0:1787757839.554905:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305567425 00000100:00000001:0.0:1787757839.554910:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757839.554914:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.554918:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757839.554927:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757839/real 1787757839] req@ffff8ffb45c31180 x1874588659688448/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757839.554939:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757839.554941:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757839.554945:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757839.554947:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757839.554950:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554952:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757839.554954:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757839.554959:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659688448/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757839.554977:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554980:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757839.554983:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554985:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757839.554986:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.554988:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757839.555001:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757839.555004:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757839.555008:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757839.555010:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757839.555014:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688448:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757839.555019:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.555021:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757839.555023:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757839.555025:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757839.555029:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757839.555031:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757839.555033:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757839.555036:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757839.555038:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757839.555040:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757839.555042:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.555044:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757839.555046:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757839.555047:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757839.555050:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757839.555051:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757839.555053:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757839.555054:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757839.555056:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757839.555058:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757839.555059:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.555061:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757839.555062:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757839.555064:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757839.555067:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757839.555069:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757839.555070:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757839.555072:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757839.555073:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757839.555075:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757839.555077:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.555079:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757839.555084:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757839.555086:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757839.555088:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757839.555090:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757840.129297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757840.129306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757840.129309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757840.129316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757840.129320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757840.129325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757840.129328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757840.129331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757840.129335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757840.321343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757840.321353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757840.321363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757840.321366:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757840.321454:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757840.321460:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757840.321465:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757841.153276:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757841.153284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757841.153287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757841.153293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757841.153298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757841.153303:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757841.153306:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757841.153309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757841.153312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757841.345307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757841.345315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757841.345323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757841.345326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757841.345355:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757841.345381:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757841.345385:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757842.177308:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757842.177317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757842.177319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757842.177326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757842.177330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757842.177385:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757842.177389:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757842.177392:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757842.177396:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757842.369322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757842.369330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757842.369359:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757842.369362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757842.369463:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757842.369468:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757842.369473:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757843.201300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757843.201312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757843.201317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757843.201378:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757843.201387:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757843.201393:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757843.201398:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757843.201402:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757843.201409:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757843.393350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757843.393363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757843.393376:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757843.393383:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757843.393388:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757843.393394:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757843.393399:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757843.393403:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757843.393407:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757843.393412:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757843.393451:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757843.393455:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757843.393459:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757844.225353:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757844.225365:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757844.225369:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757844.225377:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757844.225383:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757844.225389:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757844.225394:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757844.225398:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757844.225403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757844.417341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757844.417349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757844.417356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757844.417357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757844.417430:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757844.417437:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757844.417442:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757844.673354:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757844.673371:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757844.673375:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757844.673379:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757844.673386:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757844.673396:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10899 00000100:00080000:3.0:1787757844.673410:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757844.673416:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757844.673419:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757844.673425:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757844.673429:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757844.673433:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757844.673437:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757844.673440:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757844.673443:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757844.673450:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757844.673455:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757844.673459:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757844.673464:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757844.673469:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673472:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757844.673475:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757844.673489:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757844.673492:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673497:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757844.673500:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757844.673507:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673522:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757844.673535:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659688576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757844.673550:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757844.673557:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673585:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757844.673590:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757844.673595:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757844.673597:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757844.673600:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757844.673604:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757844.673608:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10899 00000100:00080000:3.0:1787757844.673617:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757844.673620:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757844.673624:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757844.673628:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757844.673631:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757844.673634:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757844.673638:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757844.673640:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757844.673642:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757844.673651:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757844.673654:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787757844.673656:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757844.673665:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757844.673666:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757844.673674:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787757844.673675:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757844.673685:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757844.673686:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757844.673698:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.673701:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757844.673703:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:2.0:1787757844.673713:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:3.0:1787757844.673714:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757844.673724:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673726:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757844.673735:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757844.673738:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.673745:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673746:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757844.673751:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757844.673756:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757844.673765:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659688576:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757844.673768:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.673782:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:3.0:1787757844.673784:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659688704/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757844.673791:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757844.673800:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673801:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757844.673809:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.673811:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757844.673815:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757844.673821:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757844.673826:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:2.0:1787757844.673827:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673829:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757844.673833:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757844.673839:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757844.673842:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10899 00000100:00080000:3.0:1787757844.673848:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757844.673861:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757844.673868:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787757844.673875:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757844.673936:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757844.673954:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757844.673961:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757844.673965:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.673970:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757844.673981:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.673985:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757844.673991:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757844.673993:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305572545 00000020:00000001:3.0:1787757844.673995:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757844.674000:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757844.674008:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757844.674015:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757844.674018:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757844.674021:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757844.674028:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757844.674032:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305572545 00000100:00080000:3.0:1787757844.674041:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757844.674049:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757844.674059:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757844.674065:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674070:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757844.674083:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757844/real 1787757844] req@ffff8ffb45c32a00 x1874588659688576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757844.674107:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757844.674110:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757844.674117:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757844.674121:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757844.674123:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674127:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757844.674129:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757844.674136:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659688576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757844.674147:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674152:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757844.674156:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674159:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757844.674161:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674164:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757844.674167:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757844.674171:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757844.674176:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757844.674181:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757844.674235:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659688576:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757844.674246:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674251:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757844.674255:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757844.674260:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787757844.674266:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.674270:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:3.0:1787757844.674278:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000020:00000001:2.0:1787757844.674280:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757844.674288:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757844.674291:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757844.674296:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757844.674329:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757844.674336:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757844.674339:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674345:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757844.674351:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674362:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757844.674362:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757844.674371:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757844.674374:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757844.674382:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00000001:3.0:1787757844.674383:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.674392:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674393:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757844.674401:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757844.674405:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757844.674409:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.674415:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757844.674420:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757844.674425:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757844.674434:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688704:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757844.674442:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757844.674447:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757844.674450:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.674454:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757844.674462:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674479:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757844.674482:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787757844.674490:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757844.674519:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757844.674530:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757844.674536:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757844.674541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757844.674545:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787757844.674546:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757844.674553:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674558:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.674559:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757844.674566:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757844.674571:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305571854 02000000:00000001:3.0:1787757844.674573:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674584:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305572545 00000100:00000001:2.0:1787757844.674590:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305571854 00000100:00000001:3.0:1787757844.674592:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674594:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305572545 00000100:00000001:2.0:1787757844.674602:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757844.674607:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757844.674610:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659688832/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757844.674612:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757844.674626:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757844/real 1787757844] req@ffff8ffb45c30700 x1874588659688704/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757844.674633:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757844.674643:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757844.674656:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757844.674659:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787757844.674660:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757844.674668:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:3.0:1787757844.674670:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787757844.674674:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787757844.674680:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757844.674681:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757844.674688:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757844.674689:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757844.674696:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757844.674697:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757844.674708:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:2.0:1787757844.674709:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659688704/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757844.674719:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10899 00000100:00080000:3.0:1787757844.674733:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757844.674737:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757844.674740:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757844.674745:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757844.674746:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757844.674757:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787757844.674761:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757844.674762:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757844.674768:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757844.674771:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:3.0:1787757844.674775:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757844.674779:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757844.674781:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757844.674786:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757844.674790:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00080000:3.0:1787757844.674794:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757844.674802:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757844.674806:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757844.674811:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757844.674815:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.674818:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757844.674822:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757844.674826:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674829:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757844.674830:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757844.674839:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00080000:2.0:1787757844.674845:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 02000000:00000001:3.0:1787757844.674846:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757844.674852:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757844.674858:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757844.674862:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757844.674864:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757844.674871:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674872:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757844.674886:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1787757844.674889:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659688960/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757844.674898:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674906:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757844.674911:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757844.674915:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787757844.674917:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787757844.674925:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.674926:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757844.674932:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757844.674933:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757844.674938:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757844.674945:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757844.674947:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757844.674951:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757844.674953:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674957:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.674963:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757844.674966:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.674971:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757844.674974:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757844.674978:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757844.674981:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757844.674983:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.674986:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757844.674989:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757844.674992:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757844.674998:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688832:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757844.675002:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757844.675006:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757844.675008:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.675010:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757844.675014:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675023:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757844.675025:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:2.0:1787757844.675031:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757844.675049:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757844.675055:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757844.675058:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757844.675059:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675062:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757844.675066:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675071:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305571851 00000100:00000001:2.0:1787757844.675074:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305572545 00000100:00000001:2.0:1787757844.675077:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305571851 00000100:00000001:2.0:1787757844.675080:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305572545 00000100:00000001:2.0:1787757844.675082:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757844.675085:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675087:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757844.675092:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757844/real 1787757844] req@ffff8ffb45c31500 x1874588659688832/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757844.675102:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757844.675103:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757844.675106:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757844.675108:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757844.675110:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675112:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757844.675114:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757844.675119:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659688832/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757844.675128:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675131:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757844.675134:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675136:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757844.675138:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675140:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757844.675152:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757844.675155:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757844.675159:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757844.675161:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757844.675166:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688832:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757844.675171:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757844.675173:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757844.675175:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.675177:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757844.675179:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757844.675182:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757844.675226:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688960:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757844.675230:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757844.675233:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757844.675235:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757844.675239:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757844.675244:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675252:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757844.675255:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:2.0:1787757844.675260:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757844.675277:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757844.675285:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757844.675290:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757844.675293:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675297:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757844.675329:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675338:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305571852 00000100:00000001:2.0:1787757844.675342:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305572545 00000100:00000001:2.0:1787757844.675346:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305571852 00000100:00000001:2.0:1787757844.675350:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305572545 00000100:00000001:2.0:1787757844.675353:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757844.675357:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675360:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757844.675367:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757844/real 1787757844] req@ffff8ffb45c30a80 x1874588659688960/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757844.675387:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757844.675388:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757844.675392:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757844.675394:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757844.675396:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675398:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757844.675400:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757844.675405:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659688960/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757844.675414:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675417:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757844.675419:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675421:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757844.675423:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675425:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757844.675437:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757844.675440:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757844.675443:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757844.675446:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757844.675450:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659688960:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757844.675455:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675458:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757844.675460:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757844.675462:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757844.675464:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757844.675466:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757844.675468:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757844.675470:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757844.675472:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757844.675475:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757844.675477:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675480:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757844.675481:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757844.675483:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757844.675486:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757844.675488:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757844.675489:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757844.675491:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757844.675492:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757844.675494:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757844.675496:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675498:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757844.675502:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757844.675504:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757844.675506:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757844.675509:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757845.249324:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757845.249334:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757845.249337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757845.249343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757845.249348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757845.249352:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757845.249355:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757845.249358:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757845.249362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757845.441386:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757845.441396:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757845.441399:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757845.441404:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757845.441410:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757845.441412:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757845.441415:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757845.441418:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757845.441423:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757845.441425:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757845.441454:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757845.441457:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757845.441461:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757846.273337:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757846.273345:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757846.273348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757846.273355:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757846.273360:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757846.273364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757846.273368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757846.273371:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757846.273375:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757846.465364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757846.465374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757846.465382:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757846.465385:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757846.465476:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757846.465482:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757846.465487:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757847.297282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757847.297291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757847.297295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757847.297302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757847.297307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757847.297312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757847.297316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757847.297320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757847.297325:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757847.489337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757847.489346:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757847.489353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757847.489355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757847.489381:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757847.489384:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757847.489388:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757848.321294:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757848.321302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757848.321305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757848.321311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757848.321315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757848.321320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757848.321323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757848.321327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757848.321330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757848.513351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757848.513359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757848.513367:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757848.513369:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757848.513470:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757848.513477:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757848.513482:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757849.345300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757849.345310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757849.345313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757849.345320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757849.345325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757849.345329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757849.345333:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757849.345336:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757849.345340:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757849.537350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757849.537358:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757849.537365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757849.537367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757849.537393:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757849.537397:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757849.537400:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757849.793329:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757849.793340:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757849.793344:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757849.793348:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757849.793354:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757849.793363:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10904 00000100:00080000:3.0:1787757849.793377:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757849.793381:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757849.793384:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757849.793389:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757849.793392:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757849.793396:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757849.793399:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757849.793402:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757849.793404:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757849.793409:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757849.793413:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757849.793416:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757849.793420:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757849.793424:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757849.793426:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757849.793429:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757849.793443:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757849.793445:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757849.793449:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757849.793453:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757849.793458:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757849.793472:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757849.793485:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659689088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757849.793499:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757849.793504:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757849.793530:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757849.793536:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757849.793543:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757849.793546:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757849.793549:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757849.793554:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757849.793560:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10904 00000100:00080000:3.0:1787757849.793567:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757849.793573:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757849.793576:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757849.793581:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757849.793584:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757849.793588:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757849.793593:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757849.793598:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757849.793602:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757849.793612:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757849.793618:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757849.793620:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757849.793628:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757849.793632:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793637:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757849.793637:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757849.793647:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787757849.793648:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757849.793653:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:3.0:1787757849.793656:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793660:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757849.793662:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793667:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757849.793673:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757849.793676:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757849.793679:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757849.793683:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757849.793685:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757849.793692:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757849.793694:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757849.793697:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757849.793707:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659689088:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757849.793710:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793721:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:3.0:1787757849.793728:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659689216/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757849.793730:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757849.793736:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757849.793746:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757849.793755:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787757849.793757:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757849.793759:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757849.793769:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757849.793774:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757849.793779:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757849.793783:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757849.793783:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757849.793790:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00080000:3.0:1787757849.793791:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757849.793801:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757849.793802:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757849.793810:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10904 00000100:00080000:3.0:1787757849.793817:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757849.793819:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757849.793821:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757849.793824:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757849.793826:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757849.793829:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757849.793832:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757849.793835:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757849.793839:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757849.793847:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757849.793852:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787757849.793858:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757849.793862:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757849.793872:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757849.793876:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757849.793879:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793886:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757849.793888:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757849.793891:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757849.793897:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757849.793898:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00000001:3.0:1787757849.793909:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793910:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757849.793920:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793923:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305577665 02000000:00000001:3.0:1787757849.793937:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757849.793941:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793942:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757849.793945:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757849.793950:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787757849.793954:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305577665 02000000:00000001:3.0:1787757849.793955:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793966:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757849.793972:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793973:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757849.793982:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.793986:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757849.793993:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659689344/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757849.794000:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757849/real 1787757849] req@ffff8ffb45c33b80 x1874588659689088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757849.794032:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757849.794038:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757849.794043:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757849.794054:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000400:00000001:2.0:1787757849.794061:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:3.0:1787757849.794066:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000400:00000001:2.0:1787757849.794067:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757849.794074:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787757849.794077:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:3.0:1787757849.794079:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757849.794084:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757849.794088:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787757849.794090:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794098:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757849.794099:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10904 00000100:00000001:2.0:1787757849.794106:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757849.794112:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757849.794120:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659689088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757849.794121:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757849.794126:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757849.794136:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757849.794141:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757849.794146:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757849.794155:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757849.794156:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757849.794167:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757849.794169:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757849.794170:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787757849.794182:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757849.794256:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757849.794261:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757849.794264:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757849.794269:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757849.794272:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757849.794275:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794278:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757849.794281:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757849.794284:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000008:00100000:3.0:1787757849.794287:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787757849.794290:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00000001:3.0:1787757849.794295:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757849.794299:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757849.794301:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757849.794316:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:3.0:1787757849.794317:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757849.794325:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794326:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757849.794330:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757849.794335:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:2.0:1787757849.794336:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659689088:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757849.794340:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794351:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757849.794355:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794357:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757849.794364:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757849.794369:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00100000:3.0:1787757849.794373:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659689472/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787757849.794377:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757849.794384:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757849.794389:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757849.794396:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757849.794402:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757849.794406:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757849.794409:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757849.794413:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794415:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757849.794420:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794422:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794435:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757849.794437:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794441:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757849.794443:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757849.794446:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757849.794448:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757849.794451:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757849.794454:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757849.794457:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757849.794460:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757849.794466:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689216:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757849.794470:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757849.794473:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757849.794475:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757849.794477:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757849.794482:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794489:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757849.794491:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:2.0:1787757849.794496:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757849.794513:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757849.794519:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757849.794522:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757849.794524:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794526:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757849.794530:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794535:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305576974 00000100:00000001:2.0:1787757849.794539:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305577665 00000100:00000001:2.0:1787757849.794542:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305576974 00000100:00000001:2.0:1787757849.794545:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305577665 00000100:00000001:2.0:1787757849.794548:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757849.794551:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794553:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757849.794559:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757849/real 1787757849] req@ffff8ffb45c31c00 x1874588659689216/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757849.794570:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757849.794572:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757849.794574:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757849.794576:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757849.794578:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794580:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757849.794582:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757849.794587:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659689216/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757849.794596:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794598:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757849.794601:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794603:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757849.794604:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794607:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757849.794621:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757849.794625:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757849.794629:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757849.794631:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757849.794635:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689216:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787757849.794640:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757849.794642:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757849.794643:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757849.794646:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757849.794648:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757849.794651:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757849.794655:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689344:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757849.794659:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757849.794661:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757849.794663:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757849.794665:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757849.794668:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794673:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757849.794675:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:2.0:1787757849.794680:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757849.794692:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757849.794696:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757849.794699:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757849.794701:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794703:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757849.794706:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794710:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305576971 00000100:00000001:2.0:1787757849.794713:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305577665 00000100:00000001:2.0:1787757849.794716:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305576971 00000100:00000001:2.0:1787757849.794719:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305577665 00000100:00000001:2.0:1787757849.794721:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757849.794724:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794726:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757849.794730:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757849/real 1787757849] req@ffff8ffb45c30e00 x1874588659689344/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757849.794739:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757849.794741:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757849.794743:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757849.794745:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757849.794746:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794748:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757849.794750:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757849.794755:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659689344/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757849.794764:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794767:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757849.794770:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794772:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757849.794773:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794775:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757849.794787:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757849.794790:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757849.794794:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757849.794796:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757849.794800:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689344:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757849.794804:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757849.794806:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787757849.794808:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757849.794811:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757849.794813:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757849.794815:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757849.794819:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689472:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757849.794822:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757849.794823:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757849.794825:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757849.794827:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757849.794830:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794835:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757849.794837:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:2.0:1787757849.794841:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787757849.794852:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787757849.794856:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757849.794858:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757849.794861:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794863:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757849.794866:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794870:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305576971 00000100:00000001:2.0:1787757849.794873:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305577665 00000100:00000001:2.0:1787757849.794876:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305576971 00000100:00000001:2.0:1787757849.794878:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305577665 00000100:00000001:2.0:1787757849.794881:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757849.794883:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.794885:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787757849.794890:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757849/real 1787757849] req@ffff8ffb45c30a80 x1874588659689472/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787757849.794898:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787757849.794900:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757849.794902:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787757849.794904:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757849.794906:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794908:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757849.794910:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787757849.794914:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659689472/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757849.794922:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794924:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757849.794927:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794929:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787757849.794930:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794932:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787757849.794944:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757849.794946:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757849.794950:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787757849.794952:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787757849.794957:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689472:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787757849.794961:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794964:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757849.794965:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757849.794967:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757849.794970:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757849.794971:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757849.794973:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757849.794975:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757849.794977:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757849.794979:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757849.794981:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.794983:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757849.794985:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757849.794986:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757849.794989:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757849.794990:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757849.794992:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757849.794993:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757849.794995:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757849.794997:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757849.794998:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.795000:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757849.795001:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757849.795003:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787757849.795004:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787757849.795006:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787757849.795007:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757849.795009:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757849.795010:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757849.795012:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787757849.795013:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.795015:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757849.795019:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787757849.795020:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757849.795023:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757849.795025:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757850.369315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757850.369325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757850.369328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757850.369334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757850.369339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757850.369344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757850.369347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757850.369351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757850.369354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757850.561311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757850.561320:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757850.561328:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757850.561330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757850.561376:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757850.561381:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757850.561386:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757851.393325:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757851.393334:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757851.393337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757851.393344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757851.393348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757851.393352:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757851.393356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757851.393360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757851.393363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757851.585318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757851.585330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757851.585339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757851.585342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757851.585379:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757851.585383:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757851.585388:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757852.417310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757852.417319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757852.417322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757852.417329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757852.417334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757852.417339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757852.417342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757852.417346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757852.417349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757852.609358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757852.609369:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757852.609377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757852.609379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757852.609475:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757852.609484:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757852.609493:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757853.441378:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757853.441388:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757853.441392:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757853.441399:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757853.441403:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757853.441408:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757853.441411:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757853.441430:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757853.441434:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757853.633377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757853.633388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757853.633397:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757853.633403:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757853.633407:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757853.633412:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757853.633417:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757853.633420:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757853.633423:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757853.633426:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757853.633459:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757853.633463:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757853.633467:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757854.465300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757854.465309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757854.465312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757854.465318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757854.465323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757854.465328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757854.465330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757854.465333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757854.465337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757854.657351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757854.657363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757854.657372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757854.657374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757854.657409:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757854.657413:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757854.657417:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757854.913399:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757854.913415:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757854.913419:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757854.913425:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757854.913432:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757854.913444:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10909 00000100:00080000:2.0:1787757854.913462:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757854.913467:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757854.913470:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757854.913476:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757854.913481:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757854.913485:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757854.913489:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757854.913491:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757854.913493:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757854.913500:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757854.913504:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757854.913508:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757854.913514:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757854.913519:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.913522:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757854.913525:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757854.913544:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757854.913547:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.913552:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757854.913556:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757854.913563:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.913583:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757854.913598:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659689600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757854.913614:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757854.913621:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.913655:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757854.913660:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757854.913669:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757854.913671:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757854.913673:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757854.913677:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757854.913681:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10909 00000100:00080000:2.0:1787757854.913685:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757854.913687:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757854.913690:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757854.913694:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757854.913698:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757854.913702:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757854.913706:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757854.913709:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757854.913711:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757854.913715:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757854.913718:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757854.913720:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757854.913727:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757854.913730:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757854.913741:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.913743:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757854.913746:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757854.913749:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:2.0:1787757854.913754:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757854.913755:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757854.913770:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757854.913770:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.913792:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.913794:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757854.913800:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757854.913803:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757854.913805:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.913821:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.913821:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757854.913828:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757854.913830:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.913838:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757854.913863:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659689728/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757854.913870:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659689600:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757854.913899:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757854.913910:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757854.913918:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.913926:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757854.913938:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.913943:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757854.913953:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.913966:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757854.913967:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.913971:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00080000:2.0:1787757854.913981:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757854.913982:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787757854.913995:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757854.913999:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757854.914003:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757854.914009:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757854.914015:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10909 00000100:00080000:2.0:1787757854.914026:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757854.914030:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757854.914032:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757854.914043:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000800:00000100:0.0:1787757854.914045:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787757854.914052:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757854.914058:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757854.914065:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757854.914068:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757854.914069:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757854.914076:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757854.914079:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:2.0:1787757854.914087:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757854.914089:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.914098:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914099:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787757854.914109:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757854.914115:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757854.914116:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757854.914158:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914161:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305582785 00000100:00000001:2.0:1787757854.914165:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757854.914176:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757854.914180:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787757854.914239:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305582786 00000100:00000001:0.0:1787757854.914248:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757854.914254:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757854.914259:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914264:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00000001:2.0:1787757854.914269:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.914278:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757854.914281:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757854/real 1787757854] req@ffff8ffb47a38000 x1874588659689600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757854.914291:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757854.914298:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.914313:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757854.914324:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757854.914332:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757854.914337:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757854.914341:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757854.914346:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914350:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757854.914359:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00100000:2.0:1787757854.914364:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659689856/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757854.914365:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.914385:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757854.914397:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757854.914398:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757854.914406:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.914417:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757854.914422:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757854.914433:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757854.914435:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659689600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757854.914437:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757854.914442:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757854.914450:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757854.914457:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10909 00000100:00080000:2.0:1787757854.914467:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757854.914478:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757854.914487:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757854.914488:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757854.914513:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757854.914516:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:2.0:1787757854.914519:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757854.914533:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757854.914533:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757854.914551:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:2.0:1787757854.914553:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757854.914566:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757854.914566:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757854.914581:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757854.914582:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757854.914592:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787757854.914595:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757854.914606:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757854.914607:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:2.0:1787757854.914614:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00080000:0.0:1787757854.914621:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:2.0:1787757854.914622:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757854.914633:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:2.0:1787757854.914635:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.914647:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787757854.914651:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659689600:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1787757854.914652:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757854.914676:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914680:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757854.914683:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914687:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757854.914694:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787757854.914694:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757854.914713:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.914715:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:2.0:1787757854.914720:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757854.914724:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757854.914727:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:2.0:1787757854.914729:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757854.914734:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:2.0:1787757854.914738:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757854.914739:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757854.914755:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00100000:2.0:1787757854.914757:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659689984/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757854.914760:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757854.914768:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757854.914776:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757854.914794:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757854.914795:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757854.914807:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757854.914814:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914816:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757854.914820:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914828:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757854.914832:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757854.914837:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757854.914841:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757854.914843:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.914846:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757854.914850:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757854.914854:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757854.914865:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689728:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757854.914871:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757854.914875:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757854.914877:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.914879:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757854.914885:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914895:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757854.914897:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:0.0:1787757854.914903:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757854.914926:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757854.914933:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757854.914937:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757854.914939:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914941:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757854.914946:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914952:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305582094 00000100:00000001:0.0:1787757854.914955:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305582785 00000100:00000001:0.0:1787757854.914959:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305582094 00000100:00000001:0.0:1787757854.914961:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305582786 00000100:00000001:0.0:1787757854.914964:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757854.914967:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.914969:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757854.914975:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757854/real 1787757854] req@ffff8ffb47a38380 x1874588659689728/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757854.914995:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757854.914996:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757854.915000:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757854.915002:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757854.915004:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915006:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757854.915008:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757854.915013:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659689728/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757854.915022:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915024:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757854.915027:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915029:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757854.915031:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915033:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757854.915048:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757854.915051:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757854.915055:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757854.915057:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757854.915061:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689728:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757854.915066:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757854.915068:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757854.915070:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.915074:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757854.915076:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757854.915078:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757854.915082:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689856:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757854.915086:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757854.915088:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757854.915091:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.915093:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757854.915097:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915103:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757854.915104:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757854.915109:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757854.915147:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757854.915154:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757854.915157:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757854.915159:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915161:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757854.915165:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915170:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305582091 00000100:00000001:0.0:1787757854.915174:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305582785 00000100:00000001:0.0:1787757854.915177:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305582091 00000100:00000001:0.0:1787757854.915180:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305582786 00000100:00000001:0.0:1787757854.915213:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757854.915216:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915219:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757854.915225:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757854/real 1787757854] req@ffff8ffb47a38a80 x1874588659689856/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757854.915236:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757854.915238:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757854.915242:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757854.915244:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757854.915246:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915248:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757854.915250:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757854.915255:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659689856/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757854.915263:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915266:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757854.915270:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915272:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757854.915274:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915276:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757854.915290:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757854.915293:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757854.915297:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757854.915300:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757854.915304:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689856:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757854.915309:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757854.915311:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757854.915313:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.915316:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757854.915318:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757854.915321:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757854.915325:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689984:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757854.915329:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757854.915332:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757854.915334:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757854.915337:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757854.915341:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915347:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757854.915349:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757854.915353:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757854.915373:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757854.915381:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757854.915386:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757854.915389:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915392:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757854.915397:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915404:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305582092 00000100:00000001:0.0:1787757854.915409:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305582785 00000100:00000001:0.0:1787757854.915414:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305582092 00000100:00000001:0.0:1787757854.915418:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305582786 00000100:00000001:0.0:1787757854.915423:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757854.915427:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915431:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757854.915441:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757854/real 1787757854] req@ffff8ffb47a3a680 x1874588659689984/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757854.915458:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757854.915462:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757854.915468:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757854.915471:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757854.915475:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915479:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757854.915481:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757854.915486:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659689984/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757854.915496:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915499:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757854.915502:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915504:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757854.915505:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915508:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757854.915521:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757854.915524:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757854.915527:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757854.915530:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757854.915534:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659689984:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757854.915539:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915542:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757854.915544:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757854.915547:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757854.915550:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757854.915552:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757854.915554:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757854.915557:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757854.915559:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757854.915562:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757854.915564:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915566:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757854.915567:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757854.915570:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757854.915573:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757854.915574:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757854.915575:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757854.915578:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757854.915579:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757854.915581:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757854.915583:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915584:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757854.915586:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757854.915587:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757854.915590:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757854.915591:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757854.915593:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757854.915595:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757854.915596:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757854.915600:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757854.915601:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915603:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757854.915608:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757854.915610:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757854.915613:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757854.915615:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757855.489298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757855.489307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757855.489310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757855.489316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757855.489321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757855.489325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757855.489329:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757855.489332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757855.489335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757855.681415:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757855.681427:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757855.681431:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757855.681440:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757855.681447:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757855.681450:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757855.681453:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757855.681456:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757855.681462:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757855.681464:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757855.681491:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757855.681494:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757855.681497:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757856.513295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757856.513304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757856.513307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757856.513313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757856.513318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757856.513324:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757856.513328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757856.513332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757856.513335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757856.705336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757856.705345:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757856.705352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757856.705354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757856.705436:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757856.705442:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757856.705447:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757857.537291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757857.537301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757857.537304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757857.537310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757857.537315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757857.537320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757857.537323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757857.537326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757857.537329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757857.729300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757857.729309:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757857.729319:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757857.729321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757857.729354:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757857.729357:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757857.729375:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757858.561273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757858.561283:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757858.561286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757858.561293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757858.561297:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757858.561302:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757858.561305:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757858.561308:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757858.561312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757858.753317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757858.753326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757858.753336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757858.753339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757858.753390:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757858.753394:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757858.753398:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757859.585294:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757859.585303:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757859.585306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757859.585312:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757859.585317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757859.585322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757859.585326:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757859.585329:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757859.585332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757859.777333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757859.777344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757859.777352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757859.777354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757859.777384:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757859.777388:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757859.777393:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757860.033400:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757860.033413:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757860.033416:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757860.033420:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757860.033426:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757860.033436:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10914 00000100:00080000:1.0:1787757860.033450:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757860.033455:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757860.033458:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757860.033463:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757860.033466:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757860.033470:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757860.033473:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757860.033476:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757860.033478:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757860.033484:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757860.033487:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757860.033490:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757860.033495:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757860.033499:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033501:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757860.033504:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757860.033518:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757860.033520:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033524:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757860.033527:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757860.033533:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033546:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757860.033558:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659690112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757860.033572:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757860.033577:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033603:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757860.033608:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757860.033612:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757860.033614:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757860.033616:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757860.033620:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757860.033624:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10914 00000100:00080000:1.0:1787757860.033628:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757860.033631:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757860.033633:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757860.033636:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757860.033637:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757860.033640:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757860.033642:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757860.033645:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757860.033646:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757860.033650:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757860.033653:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757860.033655:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757860.033661:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757860.033663:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033666:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757860.033672:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757860.033677:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033680:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757860.033685:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757860.033687:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033690:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757860.033693:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757860.033695:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033702:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757860.033708:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659690240/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757860.033716:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757860.033720:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757860.033728:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033735:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.033736:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:1.0:1787757860.033746:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757860.033748:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:1.0:1787757860.033752:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757860.033757:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757860.033758:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:1.0:1787757860.033760:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757860.033770:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757860.033771:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757860.033776:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10914 00000100:00080000:1.0:1787757860.033791:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757860.033794:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757860.033797:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757860.033798:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757860.033807:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757860.033810:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787757860.033812:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787757860.033814:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757860.033823:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757860.033830:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757860.033831:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033833:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757860.033852:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757860.033859:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757860.033861:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659690112:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757860.033867:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757860.033876:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757860.033879:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757860.033885:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033889:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757860.033892:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:1.0:1787757860.033893:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757860.033902:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787757860.033903:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757860.033931:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757860.033933:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757860.033949:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757860.033954:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757860.033955:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033958:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757860.033962:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757860.033966:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.033973:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757860.033981:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659690368/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757860.033989:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757860.033998:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:1.0:1787757860.033999:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757860.034011:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757860.034012:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.034027:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757860.034063:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757860.034072:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757860.034075:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757860.034080:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757860.034085:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757860.034094:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10914 00000100:00080000:1.0:1787757860.034104:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757860.034109:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757860.034112:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757860.034117:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757860.034122:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757860.034126:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757860.034131:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757860.034135:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757860.034138:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757860.034142:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757860.034153:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757860.034159:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305587905 00000100:00000001:1.0:1787757860.034163:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757860.034172:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757860.034179:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757860.034253:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.034257:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757860.034262:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757860.034266:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.034269:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034271:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:1.0:1787757860.034278:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757860.034284:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034288:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305587906 00000100:00000001:1.0:1787757860.034289:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757860.034294:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757860.034302:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034306:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757860.034314:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034315:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757860.034331:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757860.034342:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659690496/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757860.034344:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034358:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757860.034377:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757860.034380:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034383:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757860.034385:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034389:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787757860.034395:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757860.034401:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757860.034404:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757860.034407:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757860.034409:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757860.034414:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757860.034420:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757860.034425:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757860.034435:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690240:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757860.034440:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757860.034444:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757860.034447:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757860.034450:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757860.034457:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034469:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757860.034471:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:0.0:1787757860.034479:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787757860.034499:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034505:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305587214 00000100:00000001:0.0:1787757860.034509:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305587905 00000100:00000001:0.0:1787757860.034512:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305587214 00000100:00000001:0.0:1787757860.034515:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305587906 00000100:00000001:0.0:1787757860.034518:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757860.034520:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034523:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757860.034525:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757860.034527:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757860.034529:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757860.034531:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757860.034533:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757860.034537:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757860.034541:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757860.034543:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757860.034545:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757860.034548:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757860.034551:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034557:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757860.034559:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:0.0:1787757860.034564:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787757860.034572:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034576:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305587211 00000100:00000001:0.0:1787757860.034579:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305587905 00000100:00000001:0.0:1787757860.034582:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305587211 00000100:00000001:0.0:1787757860.034585:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305587906 00000100:00000001:0.0:1787757860.034588:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757860.034591:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034593:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757860.034596:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757860.034598:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757860.034601:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757860.034603:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757860.034605:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757860.034609:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690496:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757860.034612:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757860.034614:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757860.034616:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757860.034618:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757860.034621:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034627:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757860.034628:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787757860.034632:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787757860.034641:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034644:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305587211 00000100:00000001:0.0:1787757860.034647:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305587905 00000100:00000001:0.0:1787757860.034649:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305587211 00000100:00000001:0.0:1787757860.034651:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305587906 00000100:00000001:0.0:1787757860.034654:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757860.034656:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034658:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034660:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757860.034664:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757860.034666:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787757860.034668:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757860.034669:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757860.034671:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757860.034674:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757860.609260:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757860.609272:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757860.609276:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757860.609286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757860.609292:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757860.609299:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757860.609304:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757860.609309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757860.609314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757860.801344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757860.801357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757860.801369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757860.801374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757860.801470:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757860.801478:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757860.801490:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757861.633316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757861.633326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757861.633329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757861.633336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757861.633340:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757861.633346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757861.633349:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757861.633353:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757861.633356:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757861.825308:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757861.825317:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757861.825324:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757861.825325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757861.825352:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757861.825355:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757861.825359:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757862.657305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757862.657317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757862.657321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757862.657329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757862.657334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757862.657341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757862.657345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757862.657349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757862.657353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757862.849331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757862.849342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757862.849350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757862.849352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757862.849382:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757862.849386:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757862.849390:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787757863.105536:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787757863.105607:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787757863.105635:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757863.105658:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757863.105695:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757863.105701:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757863.105706:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757863.105726:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757863.105739:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757863.105751:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757863.105759:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757863.105760:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757863.105765:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757863.105776:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757863.105783:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000800:00000100:3.0:1787757863.105786:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787757863.105810:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:0.0:1787757863.105812:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757860/real 1787757863] req@ffff8ffb4a536300 x1874588659690240/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757863.105820:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757863.105826:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757863.105836:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787757863.105857:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000400:00000001:0.0:1787757863.105875:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757863.105876:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757863.105891:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757863.105904:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757863.105909:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757863.105910:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757863.105939:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757863.105949:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757863.105954:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.105961:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757863.105965:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757863.106002:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659690240/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757863.106022:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106029:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757863.106036:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106039:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757863.106042:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106048:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757863.106080:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757863.106088:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757863.106098:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757863.106105:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757863.106112:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690240:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757863.106123:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757863.106133:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757860/real 1787757863] req@ffff8ffb4a534a80 x1874588659690368/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757863.106153:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757863.106156:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757863.106164:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757863.106168:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757863.106170:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106174:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757863.106177:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757863.106231:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659690368/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757863.106249:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106253:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757863.106258:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106262:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757863.106265:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106268:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757863.106310:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757863.106317:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757863.106327:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757863.106332:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757863.106340:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757863.106355:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757863.106378:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757860/real 1787757863] req@ffff8ffb4a534700 x1874588659690496/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757863.106412:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757863.106416:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757863.106424:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757863.106430:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757863.106437:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106442:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757863.106445:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757863.106454:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659690496/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757863.106468:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106472:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757863.106477:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106480:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757863.106482:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106485:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757863.106512:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757863.106529:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757863.106537:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757863.106542:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757863.106548:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690496:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757863.106556:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757863.106566:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757860/real 1787757863] req@ffff8ffb4a537800 x1874588659690112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757863.106583:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757863.106586:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757863.106595:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757863.106599:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757863.106602:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106606:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757863.106609:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757863.106617:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659690112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757863.106632:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106636:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757863.106641:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106644:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757863.106648:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106651:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757863.106655:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757863.106661:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757863.106669:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757863.106673:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757863.106681:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659690112:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757863.106690:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106696:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757863.106701:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757863.106708:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757863.106722:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757863.106730:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757863.106736:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757863.106747:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757863.106751:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757863.106758:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757863.106762:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106766:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757863.106769:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757863.106772:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757863.106776:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757863.106778:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757863.106780:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757863.106787:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757863.106789:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757863.106792:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757863.106794:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106798:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757863.106800:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757863.106803:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757863.106809:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757863.106812:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757863.106814:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757863.106818:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757863.106821:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757863.106824:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757863.106827:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106831:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757863.106833:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757863.106836:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757863.106840:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757863.106842:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757863.106845:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757863.106849:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757863.106851:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757863.106855:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757863.106858:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106862:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757863.106877:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757863.106882:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757863.106887:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757863.106891:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757863.681296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757863.681314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757863.681317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757863.681325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757863.681330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757863.681336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757863.681341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757863.681346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757863.681350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757863.873336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757863.873347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757863.873357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757863.873362:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757863.873366:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757863.873371:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757863.873376:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757863.873380:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757863.873383:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757863.873387:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757863.873429:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757863.873433:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757863.873437:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757864.705308:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757864.705320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757864.705323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757864.705332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757864.705337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757864.705343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757864.705348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757864.705355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757864.705361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757864.897332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757864.897343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757864.897351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757864.897354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757864.897460:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757864.897470:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757864.897478:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757865.153285:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757865.153298:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757865.153302:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757865.153306:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757865.153312:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757865.153321:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10919 00000100:00080000:1.0:1787757865.153336:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757865.153341:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757865.153343:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757865.153349:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757865.153353:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757865.153357:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757865.153362:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757865.153365:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757865.153367:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757865.153372:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757865.153376:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757865.153379:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757865.153384:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757865.153388:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153391:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757865.153395:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757865.153406:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757865.153408:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153413:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757865.153416:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757865.153421:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153435:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757865.153447:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659690624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757865.153461:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757865.153467:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153488:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757865.153493:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757865.153497:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757865.153499:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757865.153502:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757865.153505:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757865.153508:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10919 00000100:00080000:1.0:1787757865.153512:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757865.153515:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757865.153517:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757865.153519:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757865.153521:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757865.153524:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757865.153527:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757865.153529:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757865.153531:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757865.153534:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757865.153537:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757865.153539:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757865.153545:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757865.153548:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153551:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757865.153557:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757865.153561:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153565:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757865.153571:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757865.153574:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153578:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757865.153582:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757865.153586:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.153598:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757865.153599:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.153608:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757865.153615:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:1.0:1787757865.153619:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659690752/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757865.153620:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757865.153626:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757865.153636:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757865.153644:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757865.153658:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757865.153659:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153670:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757865.153678:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659690624:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757865.153683:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.153699:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757865.153699:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757865.153713:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757865.153714:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757865.153721:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787757865.153724:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757865.153729:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757865.153733:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757865.153743:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757865.153749:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757865.153755:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10919 00000100:00080000:1.0:1787757865.153769:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757865.153773:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757865.153776:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757865.153781:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:1.0:1787757865.153782:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757865.153793:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000020:00000001:1.0:1787757865.153793:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757865.153800:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757865.153807:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:1.0:1787757865.153809:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757865.153818:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757865.153821:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757865.153841:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757865.153849:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757865.153853:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757865.153860:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757865.153865:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153872:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000800:00000100:0.0:1787757865.153877:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1787757865.153879:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757865.153890:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153895:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.153902:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787757865.153907:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757865.153913:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787757865.153915:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.153922:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153924:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757865.153931:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757865.153934:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787757865.153975:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.153987:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.153992:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.153999:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305593025 00000100:00100000:1.0:1787757865.154012:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659690880/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757865.154021:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787757865.154037:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305593025 00000100:00000001:1.0:1787757865.154040:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757865.154053:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154055:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757865.154061:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757865.154070:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757865.154076:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757865.154079:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757865.154082:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757865.154086:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757865.154090:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757865.154098:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757865.154103:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10919 00000100:00000001:0.0:1787757865.154109:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:1.0:1787757865.154116:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757865.154124:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757865.154129:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757865.154135:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:0.0:1787757865.154139:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757865/real 1787757865] req@ffff8ffb4a535c00 x1874588659690624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:1.0:1787757865.154141:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757865.154158:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:0.0:1787757865.154171:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:1.0:1787757865.154172:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:0.0:1787757865.154180:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757865.154252:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757865.154258:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757865.154260:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757865.154263:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757865.154269:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757865.154273:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757865.154281:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757865.154286:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757865.154300:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757865.154301:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659690624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787757865.154306:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757865.154316:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757865.154329:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154335:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757865.154338:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757865.154348:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00100000:1.0:1787757865.154349:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757865.154363:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:1.0:1787757865.154365:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154375:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757865.154377:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154383:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.154393:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:1.0:1787757865.154394:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757865.154398:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:1.0:1787757865.154401:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757865.154412:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787757865.154413:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757865.154421:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00080000:0.0:1787757865.154429:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:1.0:1787757865.154430:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154443:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1787757865.154451:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757865.154461:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659690624:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1787757865.154475:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659691008/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757865.154480:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.154491:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757865.154497:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757865.154506:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:1.0:1787757865.154510:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757865.154517:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:1.0:1787757865.154520:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757865.154525:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:1.0:1787757865.154531:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757865.154533:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757865.154547:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757865.154552:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757865.154557:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757865.154560:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.154568:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.154582:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757865.154586:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154591:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757865.154594:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757865.154597:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757865.154599:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757865.154602:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757865.154607:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757865.154611:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757865.154615:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757865.154623:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690752:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757865.154630:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757865.154634:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757865.154637:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757865.154641:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757865.154647:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154659:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757865.154663:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787757865.154671:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757865.154702:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757865.154713:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757865.154718:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757865.154721:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154724:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757865.154731:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154739:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305592334 00000100:00000001:0.0:1787757865.154745:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305593025 00000100:00000001:0.0:1787757865.154750:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305592334 00000100:00000001:0.0:1787757865.154755:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305593025 00000100:00000001:0.0:1787757865.154759:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757865.154764:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.154768:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757865.154779:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757865/real 1787757865] req@ffff8ffb4a537b80 x1874588659690752/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757865.154798:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757865.154801:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757865.154807:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757865.154811:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757865.154814:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.154817:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757865.154820:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757865.154828:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659690752/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757865.154842:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.154846:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757865.154851:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.154855:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757865.154857:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.154861:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757865.154887:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757865.154894:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757865.154901:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757865.154906:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757865.154913:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690752:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757865.154920:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757865.154923:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757865.154925:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757865.154930:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757865.154933:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757865.154936:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757865.154970:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690880:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757865.154978:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757865.154983:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757865.154987:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757865.154990:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757865.154997:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155007:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757865.155010:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787757865.155017:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757865.155043:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757865.155053:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757865.155058:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757865.155061:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155064:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757865.155069:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155076:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305592331 00000100:00000001:0.0:1787757865.155081:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305593025 00000100:00000001:0.0:1787757865.155085:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305592331 00000100:00000001:0.0:1787757865.155090:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305593025 00000100:00000001:0.0:1787757865.155095:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757865.155099:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155103:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757865.155111:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757865/real 1787757865] req@ffff8ffb4a537800 x1874588659690880/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757865.155129:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757865.155131:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757865.155137:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757865.155141:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757865.155144:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155148:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757865.155151:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757865.155159:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659690880/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757865.155173:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155178:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757865.155232:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155236:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757865.155239:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155243:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757865.155264:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757865.155270:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757865.155278:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757865.155282:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757865.155289:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659690880:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757865.155297:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757865.155300:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757865.155303:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757865.155307:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757865.155311:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757865.155315:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757865.155321:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691008:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757865.155327:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757865.155331:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757865.155334:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757865.155337:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757865.155342:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155350:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757865.155352:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787757865.155357:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757865.155376:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757865.155383:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757865.155386:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757865.155389:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155391:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757865.155395:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155399:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305592332 00000100:00000001:0.0:1787757865.155403:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305593025 00000100:00000001:0.0:1787757865.155406:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305592332 00000100:00000001:0.0:1787757865.155409:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305593025 00000100:00000001:0.0:1787757865.155412:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757865.155414:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155417:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757865.155423:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757865/real 1787757865] req@ffff8ffb4a534700 x1874588659691008/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757865.155432:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757865.155433:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757865.155436:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757865.155438:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757865.155440:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155443:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757865.155445:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757865.155449:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659691008/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757865.155457:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155460:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757865.155463:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155466:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757865.155467:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155470:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757865.155489:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757865.155495:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757865.155501:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757865.155506:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757865.155512:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691008:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757865.155520:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155524:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757865.155528:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757865.155531:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757865.155536:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757865.155539:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757865.155542:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757865.155548:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757865.155551:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757865.155555:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757865.155558:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155562:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757865.155564:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757865.155567:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757865.155570:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757865.155572:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757865.155574:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757865.155578:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757865.155580:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757865.155584:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757865.155586:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155590:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757865.155592:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757865.155595:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757865.155599:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757865.155601:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757865.155603:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757865.155607:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757865.155609:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757865.155612:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757865.155615:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155619:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757865.155626:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757865.155629:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757865.155632:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757865.155635:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757865.729279:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757865.729288:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757865.729292:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757865.729298:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757865.729302:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757865.729306:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757865.729310:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757865.729313:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757865.729317:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757865.921331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757865.921343:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757865.921348:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757865.921354:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757865.921362:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757865.921365:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757865.921368:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757865.921371:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757865.921376:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757865.921379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757865.921404:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757865.921408:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757865.921412:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757866.753296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757866.753305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757866.753307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757866.753313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757866.753318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757866.753323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757866.753327:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757866.753330:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757866.753333:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757866.945317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757866.945325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757866.945333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757866.945335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757866.945363:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757866.945367:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757866.945370:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757867.777305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757867.777315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757867.777317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757867.777324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757867.777329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757867.777333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757867.777337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757867.777340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757867.777344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757867.969331:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757867.969340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757867.969348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757867.969349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757867.969390:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757867.969393:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757867.969397:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757868.801303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757868.801313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757868.801316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757868.801323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757868.801328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757868.801332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757868.801335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757868.801338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757868.801341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757868.993336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757868.993344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757868.993352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757868.993354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757868.993442:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757868.993448:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757868.993453:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757869.825297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757869.825307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757869.825310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757869.825316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757869.825321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757869.825326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757869.825329:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757869.825332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757869.825336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757870.017320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757870.017331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757870.017341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757870.017344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757870.017381:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757870.017386:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757870.017392:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757870.273383:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757870.273395:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757870.273399:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757870.273403:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757870.273408:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757870.273418:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10925 00000100:00080000:3.0:1787757870.273431:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757870.273435:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757870.273438:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757870.273443:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757870.273446:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757870.273450:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757870.273453:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757870.273456:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757870.273457:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757870.273463:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757870.273467:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757870.273470:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757870.273475:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757870.273479:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273482:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757870.273484:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757870.273496:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757870.273498:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273503:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757870.273506:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757870.273512:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273525:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757870.273537:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659691136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757870.273551:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757870.273556:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273578:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757870.273582:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757870.273587:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757870.273589:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757870.273591:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757870.273595:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757870.273598:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10925 00000100:00080000:3.0:1787757870.273602:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757870.273605:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757870.273608:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757870.273611:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757870.273613:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757870.273616:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757870.273619:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757870.273623:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757870.273626:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757870.273630:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757870.273633:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757870.273636:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757870.273641:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757870.273646:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.273647:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757870.273649:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757870.273659:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787757870.273661:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757870.273668:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.273670:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757870.273671:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.273682:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757870.273683:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757870.273690:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757870.273691:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273700:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757870.273704:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757870.273705:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757870.273708:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.273712:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757870.273717:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.273719:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757870.273731:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659691264/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757870.273734:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659691136:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757870.273752:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757870.273760:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757870.273761:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757870.273766:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273767:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.273774:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757870.273778:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757870.273784:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757870.273785:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757870.273793:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757870.273798:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757870.273801:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757870.273805:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757870.273809:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10925 00000100:00000001:0.0:1787757870.273813:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757870.273815:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757870.273819:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787757870.273823:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757870.273827:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757870.273831:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757870.273840:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757870.273842:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757870.273846:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757870.273871:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757870.273876:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757870.273880:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757870.273888:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757870.273894:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757870.273898:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757870.273905:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757870.273910:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273915:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000800:00000100:0.0:1787757870.273925:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1787757870.273926:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757870.273936:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273941:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.273943:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757870.273952:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757870.273953:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757870.273959:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.273961:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273966:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757870.273968:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757870.273971:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757870.273978:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.273979:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.273987:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.273988:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305598145 00000100:00100000:3.0:1787757870.274002:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659691392/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757870.274007:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787757870.274020:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305598145 00000100:00000001:3.0:1787757870.274022:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757870.274029:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274030:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757870.274034:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274038:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757870.274040:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757870.274047:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757870.274050:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757870.274053:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757870.274055:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757870.274061:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757870.274067:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:0.0:1787757870.274069:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757870/real 1787757870] req@ffff8ffb45c31500 x1874588659691136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.274075:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10925 00000100:00080000:3.0:1787757870.274087:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757870.274095:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757870.274100:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:0.0:1787757870.274102:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:3.0:1787757870.274105:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000400:00000001:0.0:1787757870.274108:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000020:00000001:3.0:1787757870.274110:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757870.274116:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757870.274118:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000020:00000001:3.0:1787757870.274121:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757870.274125:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757870.274127:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757870.274130:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757870.274132:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757870.274138:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757870.274143:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757870.274145:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757870.274158:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659691136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757870.274167:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757870.274176:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757870.274223:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757870.274228:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.274231:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757870.274234:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757870.274240:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.274246:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274254:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757870.274255:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757870.274260:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.274265:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757870.274268:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787757870.274268:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757870.274275:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274284:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757870.274286:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274296:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:3.0:1787757870.274302:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659691520/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757870.274304:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274315:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757870.274320:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757870.274333:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757870.274334:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757870.274342:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757870.274347:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757870.274348:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757870.274356:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757870.274363:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659691136:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757870.274371:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274374:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757870.274377:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757870.274380:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757870.274387:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757870.274390:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757870.274393:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757870.274398:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757870.274400:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757870.274404:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757870.274406:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274408:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274417:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757870.274419:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274422:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757870.274424:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757870.274426:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757870.274428:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757870.274431:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.274434:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757870.274437:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757870.274439:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757870.274444:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691264:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757870.274449:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757870.274451:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757870.274454:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.274456:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757870.274461:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274470:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757870.274471:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757870.274477:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757870.274496:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757870.274502:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757870.274505:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757870.274507:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274510:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757870.274513:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274519:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305597454 00000100:00000001:0.0:1787757870.274522:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305598145 00000100:00000001:0.0:1787757870.274526:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305597454 00000100:00000001:0.0:1787757870.274529:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305598145 00000100:00000001:0.0:1787757870.274531:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757870.274534:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274536:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757870.274542:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757870/real 1787757870] req@ffff8ffb45c30700 x1874588659691264/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757870.274553:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757870.274555:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757870.274559:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757870.274562:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757870.274565:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274568:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757870.274571:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757870.274579:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659691264/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757870.274605:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274610:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757870.274615:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274618:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757870.274621:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274624:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757870.274648:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757870.274654:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757870.274661:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757870.274666:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757870.274673:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691264:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757870.274679:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757870.274681:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757870.274683:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.274686:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757870.274688:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757870.274691:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757870.274695:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691392:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757870.274699:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757870.274702:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757870.274704:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.274706:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757870.274710:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274718:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757870.274720:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757870.274725:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757870.274743:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757870.274750:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757870.274753:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757870.274755:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274757:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757870.274761:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274765:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305597451 00000100:00000001:0.0:1787757870.274768:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305598145 00000100:00000001:0.0:1787757870.274771:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305597451 00000100:00000001:0.0:1787757870.274774:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305598145 00000100:00000001:0.0:1787757870.274777:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757870.274779:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274781:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757870.274786:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757870/real 1787757870] req@ffff8ffb45c32a00 x1874588659691392/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757870.274795:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757870.274797:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757870.274800:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757870.274802:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757870.274803:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274806:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757870.274807:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757870.274812:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659691392/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757870.274820:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274823:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757870.274825:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274827:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757870.274829:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.274831:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757870.274844:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757870.274848:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757870.274881:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757870.274884:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757870.274889:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691392:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757870.274895:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757870.274896:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757870.274898:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.274901:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757870.274904:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757870.274906:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757870.274910:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691520:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757870.274914:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757870.274916:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757870.274918:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757870.274920:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757870.274924:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274930:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757870.274932:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787757870.274937:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757870.274952:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757870.274958:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757870.274961:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757870.274963:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274965:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757870.274969:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274973:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305597451 00000100:00000001:0.0:1787757870.274976:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305598145 00000100:00000001:0.0:1787757870.274979:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305597451 00000100:00000001:0.0:1787757870.274982:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305598145 00000100:00000001:0.0:1787757870.274985:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757870.274987:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.274989:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757870.274994:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757870/real 1787757870] req@ffff8ffb45c31180 x1874588659691520/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757870.275012:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757870.275013:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757870.275016:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757870.275018:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757870.275020:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275022:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757870.275024:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757870.275029:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659691520/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757870.275037:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275040:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757870.275042:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275044:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757870.275046:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275048:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757870.275061:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757870.275064:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757870.275067:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757870.275070:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757870.275073:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691520:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757870.275078:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275080:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757870.275082:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757870.275084:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757870.275087:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757870.275089:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757870.275091:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757870.275093:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757870.275095:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757870.275098:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757870.275100:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275102:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757870.275104:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757870.275105:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757870.275107:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757870.275109:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757870.275111:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757870.275113:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757870.275114:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757870.275116:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757870.275118:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275120:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757870.275121:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757870.275122:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757870.275124:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757870.275125:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757870.275126:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757870.275128:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757870.275129:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757870.275131:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757870.275132:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275135:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757870.275139:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757870.275141:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757870.275143:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757870.275145:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757870.849342:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757870.849351:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757870.849354:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757870.849361:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757870.849366:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757870.849371:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757870.849375:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757870.849379:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757870.849382:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757871.041369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757871.041377:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757871.041386:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757871.041388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757871.041419:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757871.041422:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757871.041426:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757871.873287:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757871.873297:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757871.873300:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757871.873307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757871.873311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757871.873316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757871.873319:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757871.873322:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757871.873326:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757872.065349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757872.065357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757872.065364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757872.065366:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757872.065389:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757872.065392:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757872.065395:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757872.897323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757872.897336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757872.897339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757872.897346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757872.897352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757872.897358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757872.897362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757872.897366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757872.897370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757873.089332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757873.089344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757873.089354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757873.089356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757873.089443:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757873.089451:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757873.089456:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757873.921330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757873.921342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757873.921346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757873.921353:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757873.921359:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757873.921365:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757873.921368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757873.921372:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757873.921376:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757874.113420:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757874.113429:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757874.113436:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757874.113440:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757874.113443:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757874.113448:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757874.113452:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757874.113455:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757874.113458:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757874.113461:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757874.113487:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757874.113490:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757874.113493:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757874.945270:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757874.945280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757874.945283:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757874.945289:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757874.945294:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757874.945299:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757874.945302:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757874.945305:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757874.945309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757875.137338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757875.137349:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757875.137353:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757875.137358:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757875.137365:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757875.137369:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757875.137372:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757875.137375:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757875.137380:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757875.137383:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757875.137420:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757875.137425:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757875.137430:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757875.393351:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757875.393364:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757875.393367:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757875.393371:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757875.393377:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757875.393387:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10930 00000100:00080000:2.0:1787757875.393401:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757875.393404:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757875.393408:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757875.393412:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757875.393416:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757875.393420:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757875.393424:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757875.393426:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757875.393428:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757875.393434:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757875.393437:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757875.393440:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757875.393445:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757875.393449:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393452:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757875.393455:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757875.393468:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757875.393470:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393474:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757875.393477:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757875.393483:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393497:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757875.393508:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659691648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757875.393522:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757875.393527:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393549:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757875.393554:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757875.393559:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757875.393560:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757875.393563:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757875.393566:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757875.393569:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10930 00000100:00080000:2.0:1787757875.393573:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757875.393576:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757875.393578:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757875.393581:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757875.393582:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757875.393585:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757875.393588:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757875.393590:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757875.393592:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757875.393595:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757875.393598:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757875.393601:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757875.393608:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757875.393610:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393613:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757875.393620:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757875.393625:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393628:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757875.393634:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757875.393636:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393638:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757875.393640:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757875.393643:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.393651:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787757875.393651:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.393662:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:2.0:1787757875.393664:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659691776/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757875.393668:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757875.393675:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757875.393681:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393683:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757875.393691:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757875.393693:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757875.393697:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757875.393700:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787757875.393704:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757875.393707:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757875.393711:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757875.393716:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757875.393721:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757875.393722:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659691648:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757875.393726:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757875.393736:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787757875.393736:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10930 00000100:00080000:2.0:1787757875.393743:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787757875.393747:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787757875.393748:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757875.393755:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393756:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757875.393790:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:2.0:1787757875.393791:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757875.393797:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757875.393800:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757875.393803:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757875.393806:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393807:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757875.393811:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757875.393820:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757875.393822:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757875.393825:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757875.393829:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757875.393832:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393834:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757875.393837:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757875.393840:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00000001:2.0:1787757875.393841:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393848:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.393850:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 02000000:00000001:2.0:1787757875.393857:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757875.393863:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787757875.393863:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393872:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757875.393873:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757875.393876:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393884:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757875.393891:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659691904/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757875.393901:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757875.393904:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.393908:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757875.393912:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757875.393916:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757875.393918:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757875.393920:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757875.393923:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757875.393929:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10930 00000800:00000100:0.0:1787757875.393938:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787757875.393939:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757875.393947:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757875.393949:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757875.393953:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757875.393956:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757875.393959:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757875.393960:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:2.0:1787757875.393964:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757875.393969:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757875.393970:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757875.393973:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757875.393977:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757875.393983:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757875.393985:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787757875.393992:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757875.393997:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757875.394000:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757875.394002:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757875.394010:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394013:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305603265 00000100:00000001:2.0:1787757875.394014:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757875.394019:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757875.394024:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.394027:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394031:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757875.394033:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757875.394039:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757875.394043:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757875.394046:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757875.394047:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305603265 02000000:00000001:2.0:1787757875.394049:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394061:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757875.394063:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757875.394076:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659692032/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757875.394084:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757875.394087:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394088:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787757875.394091:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394099:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394105:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757875.394116:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757875/real 1787757875] req@ffff8ffb47a39880 x1874588659691648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757875.394132:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757875.394136:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757875.394142:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757875.394146:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757875.394148:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394152:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757875.394154:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757875.394163:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659691648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757875.394246:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394253:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757875.394260:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394264:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757875.394267:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394271:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757875.394275:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757875.394281:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757875.394289:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757875.394292:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757875.394298:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659691648:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757875.394304:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394308:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757875.394310:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757875.394313:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757875.394319:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757875.394322:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757875.394325:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757875.394329:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757875.394331:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757875.394335:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757875.394337:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394340:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394349:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757875.394352:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394354:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757875.394356:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757875.394359:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757875.394360:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757875.394362:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757875.394366:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757875.394368:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757875.394371:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757875.394375:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691776:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757875.394380:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757875.394383:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757875.394385:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757875.394388:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757875.394392:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394401:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757875.394403:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:0.0:1787757875.394409:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757875.394427:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757875.394435:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757875.394437:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757875.394439:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394441:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757875.394445:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394451:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305602574 00000100:00000001:0.0:1787757875.394454:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305603265 00000100:00000001:0.0:1787757875.394458:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305602574 00000100:00000001:0.0:1787757875.394460:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305603265 00000100:00000001:0.0:1787757875.394463:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757875.394466:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394468:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757875.394473:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757875/real 1787757875] req@ffff8ffb47a39500 x1874588659691776/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757875.394483:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757875.394485:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757875.394487:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757875.394489:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757875.394491:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394493:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757875.394496:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757875.394500:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659691776/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757875.394509:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394511:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757875.394515:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394517:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757875.394519:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394521:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757875.394535:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757875.394539:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757875.394543:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757875.394546:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757875.394550:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691776:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757875.394554:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757875.394555:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757875.394557:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757875.394559:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757875.394561:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757875.394564:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757875.394567:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691904:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757875.394571:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757875.394573:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757875.394575:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757875.394577:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757875.394581:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394586:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757875.394588:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757875.394593:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757875.394604:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757875.394609:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757875.394611:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757875.394614:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394616:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757875.394619:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394623:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305602571 00000100:00000001:0.0:1787757875.394626:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305603265 00000100:00000001:0.0:1787757875.394630:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305602571 00000100:00000001:0.0:1787757875.394633:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305603265 00000100:00000001:0.0:1787757875.394635:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757875.394638:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394640:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757875.394645:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757875/real 1787757875] req@ffff8ffb47a3a680 x1874588659691904/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757875.394653:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757875.394655:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757875.394657:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757875.394659:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757875.394660:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394662:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757875.394664:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757875.394668:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659691904/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757875.394677:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394679:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757875.394682:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394684:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757875.394686:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394688:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757875.394699:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757875.394702:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757875.394705:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757875.394708:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757875.394712:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659691904:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757875.394717:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757875.394718:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757875.394721:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757875.394723:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757875.394725:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757875.394728:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757875.394731:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692032:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757875.394734:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757875.394736:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757875.394738:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757875.394740:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757875.394743:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394748:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757875.394749:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757875.394753:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757875.394796:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757875.394805:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757875.394809:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757875.394811:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394815:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757875.394820:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394827:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305602571 00000100:00000001:0.0:1787757875.394832:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305603265 00000100:00000001:0.0:1787757875.394836:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305602571 00000100:00000001:0.0:1787757875.394841:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305603265 00000100:00000001:0.0:1787757875.394845:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757875.394849:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.394853:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757875.394863:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757875/real 1787757875] req@ffff8ffb47a38a80 x1874588659692032/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757875.394879:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757875.394882:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757875.394887:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757875.394891:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757875.394893:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394896:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757875.394898:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757875.394904:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659692032/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757875.394913:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394916:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757875.394919:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394921:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757875.394923:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394925:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757875.394938:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757875.394941:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757875.394944:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757875.394946:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757875.394950:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692032:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757875.394955:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394958:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757875.394959:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757875.394961:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757875.394965:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757875.394967:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757875.394969:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757875.394973:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757875.394975:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757875.394977:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757875.394979:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.394981:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757875.394983:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757875.394984:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757875.394986:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757875.394988:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757875.394989:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757875.394991:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757875.394992:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757875.394995:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757875.394997:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.395000:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757875.395001:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757875.395002:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757875.395004:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757875.395006:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757875.395007:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757875.395009:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757875.395010:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757875.395013:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757875.395015:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.395017:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757875.395021:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757875.395023:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757875.395025:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757875.395027:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757875.969314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757875.969327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757875.969330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757875.969338:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757875.969344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757875.969349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757875.969353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757875.969357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757875.969360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757876.161346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757876.161355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757876.161364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757876.161366:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757876.161403:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757876.161410:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757876.161414:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757876.993305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757876.993314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757876.993316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757876.993323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757876.993328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757876.993333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757876.993336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757876.993340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757876.993344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757877.186289:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757877.186301:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757877.186311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757877.186313:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757877.186380:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757877.186402:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757877.186414:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757878.017278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757878.017290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757878.017294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757878.017301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757878.017307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757878.017313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757878.017317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757878.017321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757878.017325:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757878.209362:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757878.209373:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757878.209382:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757878.209384:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757878.209415:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757878.209418:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757878.209423:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757879.041319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757879.041327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757879.041331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757879.041337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757879.041342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757879.041347:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757879.041350:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757879.041354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757879.041357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757879.233371:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757879.233387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757879.233403:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757879.233407:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757879.233461:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757879.233468:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757879.233475:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757880.065315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757880.065325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757880.065328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757880.065334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757880.065339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757880.065343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757880.065347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757880.065350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757880.065354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757880.257342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757880.257354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757880.257365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757880.257367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757880.257403:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757880.257408:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757880.257413:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757880.513297:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757880.513314:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757880.513319:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757880.513326:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757880.513335:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757880.513347:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10935 00000100:00080000:2.0:1787757880.513365:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757880.513373:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757880.513378:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757880.513386:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757880.513390:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757880.513394:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757880.513398:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757880.513401:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757880.513402:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757880.513409:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757880.513415:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757880.513419:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757880.513424:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757880.513429:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513432:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757880.513436:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757880.513448:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757880.513451:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513454:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757880.513458:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757880.513464:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513479:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757880.513493:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659692160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757880.513508:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757880.513513:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513541:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757880.513546:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757880.513552:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757880.513555:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757880.513558:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757880.513562:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757880.513566:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10935 00000100:00080000:2.0:1787757880.513573:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757880.513576:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757880.513580:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757880.513584:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757880.513586:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757880.513589:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757880.513593:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757880.513598:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757880.513601:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757880.513607:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757880.513611:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757880.513614:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757880.513619:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757880.513622:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757880.513628:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.513629:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757880.513632:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757880.513637:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757880.513643:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:2.0:1787757880.513644:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757880.513650:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787757880.513652:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513660:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.513661:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757880.513668:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757880.513698:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757880.513707:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.513708:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513721:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:3.0:1787757880.513727:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659692160:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757880.513730:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787757880.513741:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787757880.513742:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.513750:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757880.513755:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.513760:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757880.513761:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.513769:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757880.513783:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659692288/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757880.513792:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757880.513798:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:2.0:1787757880.513810:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757880.513814:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513819:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757880.513823:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757880.513828:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757880.513830:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757880.513833:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757880.513837:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757880.513840:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10935 00000100:00080000:2.0:1787757880.513846:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757880.513848:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757880.513850:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757880.513853:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757880.513855:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757880.513857:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757880.513860:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757880.513863:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757880.513864:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757880.513868:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757880.513871:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757880.513873:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757880.513877:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757880.513880:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513882:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757880.513887:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757880.513891:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513893:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757880.513900:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757880.513902:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513905:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757880.513906:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757880.513909:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.513916:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757880.513917:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757880.513928:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659692416/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757880.513938:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757880.513941:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.513945:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757880.513948:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757880.513953:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757880.513956:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757880.513958:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757880.513962:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757880.513966:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10935 00000100:00080000:2.0:1787757880.514162:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757880.514170:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757880.514176:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:3.0:1787757880.514176:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787757880.514224:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757880.514228:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757880.514232:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757880.514236:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757880.514240:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757880.514243:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757880.514251:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757880.514256:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757880.514259:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757880.514264:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757880.514268:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.514271:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757880.514276:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757880.514280:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.514284:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757880.514294:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757880.514298:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.514303:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757880.514307:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757880.514313:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514317:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757880.514327:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514329:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757880.514336:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514341:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:2.0:1787757880.514349:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659692544/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757880.514351:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514363:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305608386 00000100:00000001:2.0:1787757880.514381:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757880.514383:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757880.514390:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757880.514398:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514400:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305608386 00000100:00000001:3.0:1787757880.514413:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757880.514422:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757880.514427:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514432:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757880.514447:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757880/real 1787757880] req@ffff8ffb47a38380 x1874588659692160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757880.514471:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757880.514475:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757880.514483:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757880.514488:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757880.514491:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514496:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757880.514499:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757880.514509:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659692160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757880.514520:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514524:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757880.514529:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514532:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757880.514534:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514537:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757880.514540:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757880.514544:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757880.514549:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757880.514552:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757880.514557:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659692160:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757880.514563:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514567:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757880.514570:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757880.514574:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757880.514579:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757880.514583:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757880.514586:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757880.514592:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757880.514596:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757880.514602:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757880.514605:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514610:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514621:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757880.514624:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514629:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757880.514632:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757880.514635:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757880.514638:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757880.514642:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.514646:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757880.514650:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757880.514654:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757880.514661:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692288:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757880.514668:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757880.514698:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757880.514702:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.514706:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757880.514714:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514727:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757880.514730:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:3.0:1787757880.514738:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757880.514767:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757880.514776:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757880.514781:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757880.514784:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514787:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757880.514794:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514800:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305607694 00000100:00000001:3.0:1787757880.514804:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305608386 00000100:00000001:3.0:1787757880.514807:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305607694 00000100:00000001:3.0:1787757880.514810:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305608386 00000100:00000001:3.0:1787757880.514813:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757880.514815:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514818:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757880.514824:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757880/real 1787757880] req@ffff8ffb47a38000 x1874588659692288/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757880.514835:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757880.514837:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757880.514840:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757880.514843:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757880.514844:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514847:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757880.514849:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757880.514853:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659692288/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757880.514863:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514866:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757880.514869:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514870:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757880.514873:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.514875:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757880.514891:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757880.514895:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757880.514899:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757880.514902:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757880.514906:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692288:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757880.514911:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757880.514912:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757880.514914:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.514917:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757880.514919:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757880.514922:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757880.514925:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757880.514930:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757880.514932:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757880.514934:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.514937:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757880.514940:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514947:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757880.514949:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:3.0:1787757880.514954:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757880.514969:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757880.514974:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757880.514977:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757880.514979:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514981:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757880.514984:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.514988:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305607691 00000100:00000001:3.0:1787757880.514991:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305608386 00000100:00000001:3.0:1787757880.514994:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305607691 00000100:00000001:3.0:1787757880.514996:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305608386 00000100:00000001:3.0:1787757880.515000:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757880.515002:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.515004:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757880.515009:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757880/real 1787757880] req@ffff8ffb47a3b480 x1874588659692416/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757880.515019:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757880.515020:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757880.515023:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757880.515024:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757880.515026:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515028:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757880.515030:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757880.515035:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659692416/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757880.515043:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515045:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757880.515048:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515050:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757880.515052:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515054:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757880.515066:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757880.515069:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757880.515073:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757880.515075:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757880.515079:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757880.515084:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757880.515086:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757880.515088:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.515090:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757880.515092:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757880.515094:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757880.515098:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692544:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757880.515102:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757880.515104:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757880.515106:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757880.515108:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757880.515112:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.515116:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757880.515118:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:3.0:1787757880.515122:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757880.515133:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757880.515138:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757880.515140:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757880.515142:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.515144:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757880.515147:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.515151:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305607691 00000100:00000001:3.0:1787757880.515154:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305608386 00000100:00000001:3.0:1787757880.515157:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305607691 00000100:00000001:3.0:1787757880.515160:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305608386 00000100:00000001:3.0:1787757880.515163:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757880.515165:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.515167:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757880.515172:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757880/real 1787757880] req@ffff8ffb47a39180 x1874588659692544/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757880.515180:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757880.515182:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757880.515229:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757880.515234:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757880.515236:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515239:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757880.515241:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757880.515248:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659692544/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757880.515266:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515270:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757880.515273:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515275:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757880.515277:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515279:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757880.515291:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757880.515294:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757880.515298:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757880.515300:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757880.515304:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692544:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757880.515309:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515311:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757880.515313:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757880.515315:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757880.515319:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757880.515321:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757880.515323:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757880.515326:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757880.515327:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757880.515331:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757880.515332:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515334:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757880.515336:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757880.515338:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757880.515340:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757880.515341:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757880.515342:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757880.515344:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757880.515345:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757880.515348:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757880.515349:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515351:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757880.515353:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757880.515355:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757880.515357:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757880.515358:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757880.515360:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757880.515363:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757880.515364:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757880.515367:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757880.515369:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515371:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757880.515375:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757880.515377:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757880.515380:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757880.515382:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757881.089286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757881.089295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757881.089298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757881.089305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757881.089309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757881.089314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757881.089317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757881.089320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757881.089323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757881.281341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757881.281353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757881.281367:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757881.281371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757881.281439:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757881.281447:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757881.281454:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757882.113358:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757882.113371:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757882.113374:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757882.113382:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757882.113387:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757882.113393:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757882.113398:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757882.113402:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757882.113406:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757882.305333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757882.305342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757882.305350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757882.305351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757882.305379:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757882.305383:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757882.305386:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757883.137326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757883.137337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757883.137342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757883.137351:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757883.137356:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757883.137363:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757883.137368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757883.137373:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757883.137378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757883.329341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757883.329353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757883.329365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757883.329372:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757883.329376:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757883.329382:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757883.329388:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757883.329393:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757883.329398:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757883.329403:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757883.329438:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757883.329442:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757883.329445:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757884.161296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757884.161307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757884.161310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757884.161320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757884.161326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757884.161332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757884.161336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757884.161340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757884.161344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757884.353338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757884.353349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757884.353360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757884.353363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757884.353400:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757884.353406:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757884.353412:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757885.185306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757885.185315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757885.185318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757885.185325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757885.185330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757885.185334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757885.185337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757885.185341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757885.185344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757885.377339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757885.377352:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757885.377357:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757885.377362:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757885.377370:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757885.377374:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757885.377378:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757885.377383:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757885.377391:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757885.377394:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757885.377462:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757885.377468:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757885.377473:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757885.633411:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757885.633423:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757885.633427:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757885.633431:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757885.633437:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757885.633446:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10940 00000100:00080000:1.0:1787757885.633460:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757885.633464:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757885.633467:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757885.633472:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757885.633476:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757885.633480:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757885.633483:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757885.633486:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757885.633488:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757885.633495:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757885.633498:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757885.633501:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757885.633506:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757885.633511:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.633514:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757885.633517:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757885.633529:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757885.633532:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.633536:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757885.633539:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757885.633544:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.633558:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757885.633571:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659692672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757885.633616:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757885.633621:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.633643:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757885.633647:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757885.633651:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757885.633653:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757885.633656:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757885.633660:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757885.633663:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10940 00000100:00080000:1.0:1787757885.633668:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757885.633671:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757885.633673:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757885.633677:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757885.633680:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757885.633683:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757885.633687:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757885.633689:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757885.633692:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757885.633694:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787757885.633698:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757885.633703:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757885.633706:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.633711:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787757885.633713:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757885.633720:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787757885.633724:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757885.633726:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757885.633738:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.633741:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757885.633743:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:3.0:1787757885.633749:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:1.0:1787757885.633753:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787757885.633757:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787757885.633761:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.633766:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757885.633771:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659692672:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757885.633774:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757885.633779:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.633785:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787757885.633786:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757885.633793:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757885.633795:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757885.633798:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.633801:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.633807:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757885.633810:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.633815:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757885.633825:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659692800/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757885.633838:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757885.633841:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757885.633842:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.633849:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:1.0:1787757885.633853:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.633859:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787757885.633860:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757885.633867:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757885.633869:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757885.633872:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757885.633876:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757885.633881:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10940 00000100:00080000:1.0:1787757885.633886:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757885.633888:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757885.633890:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757885.633893:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757885.633896:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757885.633898:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757885.633901:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757885.633903:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757885.633907:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:3.0:1787757885.633914:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787757885.633921:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757885.633928:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757885.633932:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757885.633936:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787757885.633937:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757885.633945:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.633948:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757885.633949:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757885.633958:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787757885.633959:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757885.633968:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00000001:1.0:1787757885.633969:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.633976:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.633980:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757885.633983:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757885.633988:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305613505 02000000:00000001:1.0:1787757885.633989:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.634000:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757885.634002:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757885.634004:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.634013:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634014:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:1.0:1787757885.634023:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659692928/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757885.634029:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305613505 00000100:00000001:1.0:1787757885.634043:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757885.634047:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634050:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757885.634052:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757885.634063:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787757885.634065:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757885.634067:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757885.634073:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757885.634079:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757885.634081:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.634083:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757885.634094:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10940 00000100:00000001:3.0:1787757885.634094:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:1.0:1787757885.634104:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757885.634107:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757885.634108:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757885.634114:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:3.0:1787757885.634117:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757885/real 1787757885] req@ffff8ffb4a534a80 x1874588659692672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:1.0:1787757885.634118:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757885.634124:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757885.634134:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757885.634143:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757885.634147:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757885.634162:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:3.0:1787757885.634166:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787757885.634167:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787757885.634174:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:1.0:1787757885.634175:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757885.634224:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757885.634228:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.634230:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757885.634234:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757885.634237:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.634239:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634241:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:1.0:1787757885.634246:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757885.634250:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:1.0:1787757885.634252:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634256:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757885.634258:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757885.634265:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787757885.634267:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:1.0:1787757885.634270:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634273:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757885.634279:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757885.634289:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659692672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757885.634295:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659693056/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757885.634337:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757885.634340:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757885.634346:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757885.634347:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757885.634356:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634358:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634365:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757885.634367:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634371:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757885.634375:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757885.634379:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757885.634386:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757885.634392:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757885.634399:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659692672:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757885.634406:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634409:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757885.634411:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757885.634415:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757885.634419:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757885.634422:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757885.634424:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757885.634428:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757885.634431:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757885.634434:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757885.634436:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634438:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634446:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757885.634449:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634452:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757885.634455:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757885.634457:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757885.634459:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757885.634461:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.634464:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757885.634467:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757885.634470:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757885.634474:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692800:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757885.634479:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757885.634482:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757885.634484:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.634486:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757885.634490:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634498:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757885.634500:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:3.0:1787757885.634506:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757885.634525:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757885.634532:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757885.634535:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757885.634537:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634540:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757885.634543:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634549:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305612814 00000100:00000001:3.0:1787757885.634553:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305613505 00000100:00000001:3.0:1787757885.634556:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305612814 00000100:00000001:3.0:1787757885.634559:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305613505 00000100:00000001:3.0:1787757885.634561:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757885.634564:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634566:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757885.634572:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757885/real 1787757885] req@ffff8ffb4a536300 x1874588659692800/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757885.634616:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757885.634619:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757885.634624:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757885.634626:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757885.634628:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634631:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757885.634632:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757885.634639:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659692800/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757885.634648:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634651:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757885.634655:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634656:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757885.634658:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634660:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757885.634675:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757885.634678:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757885.634681:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757885.634684:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757885.634689:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692800:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757885.634694:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757885.634696:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757885.634698:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.634701:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757885.634703:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757885.634705:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757885.634709:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757885.634713:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757885.634716:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757885.634718:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.634720:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757885.634724:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634731:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757885.634733:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757885.634738:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757885.634755:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757885.634761:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757885.634764:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757885.634766:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634768:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757885.634772:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634776:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305612811 00000100:00000001:3.0:1787757885.634779:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305613505 00000100:00000001:3.0:1787757885.634782:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305612811 00000100:00000001:3.0:1787757885.634785:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305613505 00000100:00000001:3.0:1787757885.634788:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757885.634790:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634792:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757885.634797:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757885/real 1787757885] req@ffff8ffb4a534700 x1874588659692928/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757885.634815:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757885.634816:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757885.634819:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757885.634821:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757885.634823:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634825:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757885.634827:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757885.634831:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659692928/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757885.634839:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634842:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757885.634845:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634847:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757885.634849:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634851:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757885.634862:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757885.634865:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757885.634869:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757885.634871:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757885.634875:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659692928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757885.634880:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757885.634882:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757885.634884:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.634886:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757885.634888:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757885.634890:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757885.634894:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693056:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757885.634897:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757885.634899:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757885.634902:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757885.634904:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757885.634907:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634912:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757885.634914:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757885.634919:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757885.634930:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757885.634935:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757885.634937:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757885.634939:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634941:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757885.634944:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634948:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305612811 00000100:00000001:3.0:1787757885.634951:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305613505 00000100:00000001:3.0:1787757885.634954:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305612811 00000100:00000001:3.0:1787757885.634957:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305613505 00000100:00000001:3.0:1787757885.634960:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757885.634962:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.634964:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757885.634969:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757885/real 1787757885] req@ffff8ffb4a537800 x1874588659693056/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757885.634977:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757885.634979:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757885.634981:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757885.634983:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757885.634984:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.634986:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757885.634988:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757885.634992:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659693056/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757885.635000:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.635004:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757885.635010:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.635013:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757885.635016:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.635019:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757885.635039:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757885.635044:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757885.635052:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757885.635056:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757885.635063:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693056:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757885.635071:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.635075:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757885.635078:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757885.635081:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757885.635085:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757885.635088:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757885.635091:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757885.635095:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757885.635098:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757885.635102:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757885.635105:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.635109:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757885.635111:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757885.635113:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757885.635117:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757885.635120:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757885.635122:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757885.635125:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757885.635126:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757885.635129:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757885.635130:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.635132:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757885.635134:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757885.635135:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757885.635137:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757885.635138:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757885.635140:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757885.635143:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757885.635145:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757885.635147:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757885.635149:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.635151:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757885.635156:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757885.635159:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757885.635161:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757885.635164:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757886.209293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757886.209302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757886.209305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757886.209310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757886.209315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757886.209320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757886.209323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757886.209327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757886.209330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757886.401370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757886.401378:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757886.401390:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757886.401393:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757886.401422:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757886.401426:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757886.401430:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757887.233328:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757887.233337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757887.233340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757887.233347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757887.233352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757887.233357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757887.233360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757887.233363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757887.233367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757887.425355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757887.425364:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757887.425372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757887.425374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757887.425402:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757887.425405:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757887.425409:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757888.257339:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757888.257349:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757888.257353:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757888.257361:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757888.257368:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757888.257374:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757888.257379:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757888.257384:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757888.257389:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757888.449330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757888.449338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757888.449347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757888.449350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757888.449381:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757888.449384:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757888.449388:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757889.281350:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757889.281359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757889.281362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757889.281368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757889.281373:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757889.281377:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757889.281380:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757889.281383:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757889.281387:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757889.473285:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757889.473293:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757889.473301:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757889.473304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757889.473389:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757889.473393:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757889.473397:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757890.305289:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757890.305298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757890.305301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757890.305307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757890.305311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757890.305316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757890.305320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757890.305323:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757890.305326:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757890.497300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757890.497308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757890.497317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757890.497319:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757890.497350:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757890.497354:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757890.497358:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757890.753312:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757890.753324:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757890.753328:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757890.753332:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757890.753337:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757890.753346:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10945 00000100:00080000:1.0:1787757890.753359:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757890.753364:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757890.753367:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757890.753372:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757890.753375:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757890.753379:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757890.753382:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757890.753385:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757890.753387:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757890.753393:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757890.753396:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757890.753399:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757890.753403:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757890.753407:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753410:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757890.753412:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757890.753425:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757890.753427:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753431:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757890.753434:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757890.753439:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753454:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757890.753465:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659693184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757890.753480:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757890.753486:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753557:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757890.753564:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757890.753575:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757890.753578:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757890.753581:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757890.753587:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757890.753592:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10945 00000100:00080000:1.0:1787757890.753600:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757890.753606:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757890.753612:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757890.753618:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757890.753621:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757890.753626:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757890.753631:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757890.753637:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757890.753637:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757890.753641:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757890.753652:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757890.753660:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:1.0:1787757890.753663:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757890.753668:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757890.753671:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.753674:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757890.753682:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757890.753689:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757890.753691:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757890.753699:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.753701:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757890.753703:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757890.753713:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787757890.753716:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787757890.753720:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753727:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757890.753734:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757890.753740:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753744:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:3.0:1787757890.753744:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659693184:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757890.753750:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757890.753758:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753771:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.753773:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:1.0:1787757890.753786:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659693312/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787757890.753787:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757890.753798:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.753813:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757890.753827:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757890.753831:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753837:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.753837:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757890.753855:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757890.753860:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757890.753862:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757890.753865:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757890.753870:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757890.753875:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10945 00000100:00080000:1.0:1787757890.753880:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757890.753883:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757890.753884:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757890.753887:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787757890.753893:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:1.0:1787757890.753894:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757890.753901:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757890.753907:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757890.753909:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757890.753911:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757890.753915:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757890.753917:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:1.0:1787757890.753919:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757890.753928:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757890.753932:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787757890.753933:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757890.753940:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753943:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757890.753947:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757890.753951:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753954:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757890.753964:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757890.753966:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753971:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757890.753973:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757890.753976:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.753983:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757890.753991:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659693440/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757890.754002:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757890.754005:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:3.0:1787757890.754017:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757890.754018:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757890.754029:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757890.754035:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757890.754038:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757890.754042:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757890.754046:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757890.754047:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757890.754060:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10945 00000100:00000001:3.0:1787757890.754062:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787757890.754065:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757890.754069:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.754071:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757890.754080:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757890.754082:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:1.0:1787757890.754095:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757890.754100:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757890.754103:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757890.754104:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757890.754116:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757890.754120:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305618625 00000100:00000001:1.0:1787757890.754123:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757890.754132:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757890.754140:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757890.754144:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754144:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:1.0:1787757890.754156:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757890.754162:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757890.754166:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305618625 02000000:00000001:1.0:1787757890.754167:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.754180:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757890.754231:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757890.754234:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00000001:1.0:1787757890.754241:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754246:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757890.754247:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757890.754269:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757890.754270:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757890.754274:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.754281:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757890.754285:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787757890.754285:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:1.0:1787757890.754296:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.754304:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757890.754314:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757890/real 1787757890] req@ffff8ffb4a537b80 x1874588659693184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757890.754316:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659693568/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757890.754381:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757890.754396:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757890.754404:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787757890.754407:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757890.754411:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757890.754421:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757890.754426:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757890.754429:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754434:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757890.754437:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757890.754448:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659693184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757890.754460:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754466:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757890.754470:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754472:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757890.754473:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754477:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757890.754480:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757890.754485:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757890.754520:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757890.754524:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757890.754530:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659693184:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757890.754538:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754541:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757890.754544:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757890.754547:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757890.754554:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757890.754559:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757890.754562:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757890.754569:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757890.754572:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757890.754576:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757890.754578:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754581:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754592:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757890.754595:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754598:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757890.754601:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757890.754603:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757890.754606:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757890.754608:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.754611:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757890.754615:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757890.754618:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757890.754622:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693312:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757890.754627:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757890.754630:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757890.754633:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.754635:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757890.754640:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754650:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757890.754652:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:3.0:1787757890.754659:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757890.754679:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757890.754686:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757890.754689:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757890.754690:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754693:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757890.754697:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754702:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305617934 00000100:00000001:3.0:1787757890.754706:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305618625 00000100:00000001:3.0:1787757890.754710:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305617934 00000100:00000001:3.0:1787757890.754713:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305618625 00000100:00000001:3.0:1787757890.754717:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757890.754719:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754722:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757890.754727:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757890/real 1787757890] req@ffff8ffb4a535c00 x1874588659693312/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757890.754739:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757890.754741:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757890.754744:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757890.754746:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757890.754748:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754750:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757890.754752:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757890.754757:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659693312/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757890.754766:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754769:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757890.754772:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754773:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757890.754775:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754777:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757890.754799:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757890.754802:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757890.754806:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757890.754809:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757890.754813:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693312:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757890.754817:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757890.754820:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757890.754822:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.754825:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757890.754827:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757890.754830:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757890.754834:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757890.754838:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757890.754841:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757890.754843:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.754845:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757890.754849:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754855:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757890.754857:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757890.754862:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757890.754874:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757890.754879:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757890.754882:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757890.754884:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754886:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757890.754889:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754893:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305617931 00000100:00000001:3.0:1787757890.754897:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305618625 00000100:00000001:3.0:1787757890.754900:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305617931 00000100:00000001:3.0:1787757890.754903:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305618625 00000100:00000001:3.0:1787757890.754906:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757890.754908:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.754910:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757890.754915:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757890/real 1787757890] req@ffff8ffb4a537800 x1874588659693440/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757890.754924:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757890.754925:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757890.754927:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757890.754929:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757890.754931:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754933:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757890.754934:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757890.754939:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659693440/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757890.754956:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754959:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757890.754962:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754964:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757890.754966:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.754968:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757890.754986:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757890.754989:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757890.754992:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757890.754995:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757890.754998:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757890.755003:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757890.755005:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757890.755006:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.755009:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757890.755011:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757890.755013:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757890.755017:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757890.755020:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757890.755022:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757890.755024:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757890.755026:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757890.755030:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.755035:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757890.755037:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757890.755042:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757890.755053:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757890.755058:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757890.755060:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757890.755062:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.755065:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757890.755068:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.755072:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305617931 00000100:00000001:3.0:1787757890.755075:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305618625 00000100:00000001:3.0:1787757890.755078:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305617931 00000100:00000001:3.0:1787757890.755081:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305618625 00000100:00000001:3.0:1787757890.755084:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757890.755086:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.755088:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757890.755093:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757890/real 1787757890] req@ffff8ffb4a534700 x1874588659693568/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757890.755101:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757890.755102:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757890.755105:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757890.755106:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757890.755108:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755110:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757890.755112:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757890.755116:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659693568/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757890.755124:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755127:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757890.755130:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755133:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757890.755134:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755136:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757890.755155:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757890.755158:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757890.755161:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757890.755164:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757890.755168:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757890.755173:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755176:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757890.755177:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757890.755179:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757890.755215:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757890.755218:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757890.755219:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757890.755222:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757890.755223:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757890.755226:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757890.755228:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755230:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757890.755232:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757890.755234:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757890.755236:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757890.755238:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757890.755239:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757890.755243:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757890.755245:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757890.755247:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757890.755249:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755251:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757890.755252:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757890.755254:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757890.755256:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757890.755257:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757890.755259:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757890.755262:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757890.755264:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757890.755267:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757890.755270:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755274:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757890.755279:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757890.755282:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757890.755286:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757890.755289:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757891.329327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757891.329336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757891.329339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757891.329345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757891.329349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757891.329354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757891.329358:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757891.329361:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757891.329365:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757891.521387:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757891.521398:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757891.521407:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757891.521410:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757891.521453:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757891.521457:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757891.521463:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757892.353267:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757892.353273:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757892.353275:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757892.353279:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757892.353282:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757892.353285:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757892.353287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757892.353288:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757892.353290:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757892.545334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757892.545342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757892.545347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757892.545348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757892.545386:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757892.545389:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757892.545393:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757893.377286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757893.377295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757893.377298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757893.377304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757893.377309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757893.377315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757893.377318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757893.377322:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757893.377325:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757893.569305:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757893.569312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757893.569321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757893.569326:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757893.569329:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757893.569333:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757893.569337:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757893.569339:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757893.569342:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757893.569345:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757893.569409:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757893.569415:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757893.569419:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757894.401323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757894.401332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757894.401335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757894.401341:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757894.401346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757894.401351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757894.401355:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757894.401358:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757894.401362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757894.593327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757894.593335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757894.593342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757894.593345:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757894.593374:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757894.593378:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757894.593381:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757895.425298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757895.425307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757895.425309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757895.425316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757895.425321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757895.425326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757895.425329:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757895.425333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757895.425351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757895.617300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757895.617310:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757895.617314:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757895.617319:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757895.617325:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757895.617328:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757895.617331:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757895.617335:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757895.617340:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757895.617342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757895.617371:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757895.617374:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757895.617378:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757895.873308:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757895.873319:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757895.873323:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757895.873341:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757895.873348:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757895.873357:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10950 00000100:00080000:1.0:1787757895.873371:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757895.873375:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757895.873377:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757895.873383:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757895.873386:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757895.873390:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757895.873394:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757895.873397:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757895.873437:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757895.873445:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757895.873448:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757895.873452:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757895.873456:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757895.873460:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873463:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757895.873466:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757895.873480:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757895.873483:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873486:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757895.873490:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757895.873495:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873510:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757895.873522:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659693696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757895.873536:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757895.873542:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873561:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757895.873565:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757895.873570:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757895.873572:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757895.873574:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757895.873578:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757895.873581:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10950 00000100:00080000:1.0:1787757895.873586:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757895.873589:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757895.873591:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757895.873595:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757895.873597:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757895.873600:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757895.873604:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757895.873608:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757895.873610:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757895.873615:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757895.873618:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757895.873620:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757895.873625:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787757895.873627:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757895.873635:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.873636:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757895.873638:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757895.873644:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:1.0:1787757895.873647:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757895.873650:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:1.0:1787757895.873655:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.873657:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873661:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.873667:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757895.873669:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757895.873673:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757895.873675:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873686:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757895.873687:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757895.873690:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757895.873696:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757895.873701:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659693696:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757895.873705:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.873718:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:1.0:1787757895.873724:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659693824/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787757895.873727:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757895.873735:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.873744:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757895.873752:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757895.873755:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.873756:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873760:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757895.873767:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757895.873772:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757895.873774:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757895.873776:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757895.873780:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787757895.873782:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787757895.873785:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10950 00000100:00000001:3.0:1787757895.873788:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00080000:1.0:1787757895.873794:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757895.873797:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757895.873799:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757895.873805:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757895.873809:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757895.873811:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757895.873814:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757895.873818:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757895.873821:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757895.873823:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757895.873827:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757895.873830:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757895.873832:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757895.873837:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757895.873840:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873842:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757895.873845:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757895.873848:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:3.0:1787757895.873857:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757895.873858:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757895.873872:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757895.873874:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873877:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:3.0:1787757895.873880:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787757895.873881:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757895.873889:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.873892:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757895.873898:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.873899:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.873911:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:1.0:1787757895.873917:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659693952/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757895.873922:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.873934:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305623745 00000100:00000001:1.0:1787757895.873941:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757895.873946:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.873950:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.873952:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787757895.873954:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757895.873964:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757895.873967:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305623745 00000100:00000001:1.0:1787757895.873968:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757895.873972:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757895.873979:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757895.873980:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757895.873990:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10950 00000100:00000001:3.0:1787757895.873991:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757895.874000:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757895.874001:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757895.874007:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757895.874009:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757895.874014:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757895.874018:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757895.874021:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:3.0:1787757895.874024:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757895/real 1787757895] req@ffff8ffb4a536300 x1874588659693696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:1.0:1787757895.874025:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757895.874033:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757895.874041:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757895.874044:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757895.874056:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757895.874066:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757895.874073:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757895.874080:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757895.874084:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787757895.874087:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787757895.874088:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757895.874093:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:3.0:1787757895.874096:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000008:00000001:1.0:1787757895.874099:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.874106:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874112:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:1.0:1787757895.874113:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757895.874121:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.874125:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:3.0:1787757895.874125:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:1.0:1787757895.874130:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787757895.874134:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757895.874135:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874147:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757895.874148:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874151:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757895.874159:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659694080/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757895.874167:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659693696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757895.874219:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757895.874222:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757895.874226:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874267:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874273:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757895.874278:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874280:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757895.874282:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874285:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757895.874288:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757895.874292:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757895.874298:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757895.874303:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757895.874310:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659693696:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757895.874317:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874322:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757895.874325:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757895.874328:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757895.874334:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757895.874337:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757895.874355:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757895.874361:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757895.874363:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757895.874366:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757895.874368:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874371:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874380:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757895.874382:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874385:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757895.874388:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757895.874390:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757895.874393:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757895.874395:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.874398:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757895.874426:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757895.874429:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757895.874434:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693824:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757895.874439:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757895.874443:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757895.874445:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.874447:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757895.874452:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874461:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757895.874463:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:3.0:1787757895.874469:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757895.874488:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757895.874495:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757895.874498:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757895.874500:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874502:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757895.874506:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874511:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305623054 00000100:00000001:3.0:1787757895.874515:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305623745 00000100:00000001:3.0:1787757895.874519:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305623054 00000100:00000001:3.0:1787757895.874522:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305623745 00000100:00000001:3.0:1787757895.874524:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757895.874527:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874529:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757895.874535:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757895/real 1787757895] req@ffff8ffb4a534a80 x1874588659693824/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757895.874546:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757895.874548:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757895.874551:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757895.874553:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757895.874556:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874558:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757895.874560:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757895.874565:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659693824/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757895.874574:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874577:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757895.874579:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874581:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757895.874583:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874585:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757895.874601:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757895.874604:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757895.874608:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757895.874611:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757895.874615:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693824:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757895.874619:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757895.874621:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757895.874623:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.874626:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757895.874628:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757895.874631:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757895.874634:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693952:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757895.874638:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757895.874640:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757895.874642:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.874644:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757895.874647:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874654:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757895.874655:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757895.874660:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757895.874672:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757895.874677:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757895.874680:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757895.874682:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874684:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757895.874687:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874691:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305623051 00000100:00000001:3.0:1787757895.874694:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305623745 00000100:00000001:3.0:1787757895.874697:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305623051 00000100:00000001:3.0:1787757895.874699:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305623745 00000100:00000001:3.0:1787757895.874702:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757895.874704:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874706:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757895.874711:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757895/real 1787757895] req@ffff8ffb4a534700 x1874588659693952/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757895.874720:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757895.874721:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757895.874724:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757895.874726:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757895.874728:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874730:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757895.874732:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757895.874736:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659693952/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757895.874745:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874747:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757895.874750:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874752:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757895.874753:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874755:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757895.874769:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757895.874772:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757895.874776:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757895.874779:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757895.874783:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659693952:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757895.874787:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757895.874789:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757895.874790:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.874793:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757895.874795:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757895.874797:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757895.874800:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757895.874804:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757895.874805:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757895.874807:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757895.874809:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757895.874812:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874817:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757895.874820:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757895.874824:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757895.874834:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757895.874839:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757895.874841:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757895.874843:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874846:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757895.874848:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874852:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305623051 00000100:00000001:3.0:1787757895.874856:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305623745 00000100:00000001:3.0:1787757895.874859:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305623051 00000100:00000001:3.0:1787757895.874861:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305623745 00000100:00000001:3.0:1787757895.874864:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757895.874867:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.874869:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757895.874874:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757895/real 1787757895] req@ffff8ffb4a537800 x1874588659694080/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757895.874884:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757895.874885:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757895.874888:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757895.874890:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757895.874891:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874893:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757895.874895:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757895.874900:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659694080/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757895.874908:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874911:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757895.874913:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874915:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757895.874917:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874919:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757895.874933:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757895.874936:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757895.874939:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757895.874941:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757895.874945:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757895.874949:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874951:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757895.874953:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757895.874955:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757895.874957:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757895.874960:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757895.874961:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757895.874964:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757895.874966:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757895.874968:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757895.874971:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874973:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757895.874974:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757895.874976:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757895.874978:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757895.874980:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757895.874981:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757895.874984:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757895.874985:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757895.874987:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757895.874988:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.874990:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757895.874992:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757895.874993:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757895.874995:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757895.874996:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757895.874997:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757895.874999:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757895.875001:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757895.875003:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757895.875004:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.875006:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757895.875010:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757895.875012:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757895.875015:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757895.875017:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757896.449347:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757896.449371:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757896.449376:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757896.449385:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757896.449434:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757896.449440:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757896.449445:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757896.449449:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757896.449454:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757896.641428:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757896.641438:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757896.641450:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757896.641453:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757896.641495:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757896.641502:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757896.641510:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757897.473250:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757897.473255:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757897.473257:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757897.473261:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757897.473264:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757897.473267:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757897.473269:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757897.473271:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757897.473273:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757897.665258:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757897.665263:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757897.665267:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757897.665268:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757897.665323:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757897.665326:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757897.665329:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757898.497323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757898.497344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757898.497347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757898.497375:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757898.497380:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757898.497396:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757898.497400:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757898.497403:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757898.497407:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757898.689286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757898.689295:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757898.689307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757898.689311:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757898.689348:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757898.689381:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757898.689386:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757899.521266:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757899.521275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757899.521278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757899.521283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757899.521288:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757899.521292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757899.521295:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757899.521298:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757899.521302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757899.713295:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757899.713304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757899.713358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757899.713361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757899.713388:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757899.713392:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757899.713396:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757900.545281:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757900.545291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757900.545294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757900.545301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757900.545306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757900.545310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757900.545313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757900.545316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757900.545356:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757900.737348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757900.737356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757900.737364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757900.737365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757900.737452:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757900.737458:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757900.737463:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757900.993292:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757900.993305:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757900.993308:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757900.993365:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757900.993373:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757900.993385:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10955 00000100:00080000:1.0:1787757900.993402:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757900.993408:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757900.993412:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757900.993419:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757900.993423:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757900.993429:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757900.993434:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757900.993438:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757900.993441:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757900.993451:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757900.993456:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757900.993461:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757900.993467:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757900.993471:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993474:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757900.993477:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757900.993490:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757900.993493:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993496:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757900.993500:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757900.993505:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993519:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757900.993533:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659694208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757900.993547:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757900.993552:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993573:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757900.993578:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757900.993583:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757900.993585:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757900.993587:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757900.993591:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757900.993594:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10955 00000100:00080000:1.0:1787757900.993599:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757900.993603:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757900.993604:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757900.993608:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757900.993610:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757900.993614:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757900.993617:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757900.993620:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757900.993623:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757900.993628:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757900.993632:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757900.993634:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757900.993638:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787757900.993641:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757900.993646:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787757900.993648:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.993653:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757900.993654:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757900.993659:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00100000:1.0:1787757900.993665:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757900.993666:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787757900.993677:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.993679:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757900.993681:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.993686:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757900.993689:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757900.993694:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757900.993696:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993705:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757900.993709:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:3.0:1787757900.993711:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659694208:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757900.993713:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.993725:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787757900.993727:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.993732:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757900.993737:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757900.993740:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659694336/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787757900.993743:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757900.993758:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993759:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757900.993766:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993772:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757900.993775:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757900.993779:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757900.993781:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757900.993783:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757900.993787:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787757900.993791:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787757900.993792:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10955 00000100:00000001:3.0:1787757900.993801:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00080000:1.0:1787757900.993802:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757900.993809:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757900.993814:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757900.993816:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:1.0:1787757900.993818:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757900.993825:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757900.993827:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757900.993831:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757900.993833:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757900.993835:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757900.993839:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757900.993843:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757900.993846:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757900.993849:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757900.993852:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993854:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757900.993858:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757900.993862:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993865:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757900.993871:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757900.993872:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993875:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757900.993877:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757900.993882:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:3.0:1787757900.993884:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757900.993891:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.993902:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:1.0:1787757900.993903:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659694464/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757900.993913:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757900.993919:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993921:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757900.993926:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757900.993928:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.993935:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.993936:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757900.993946:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787757900.993947:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305628865 00000100:00080000:1.0:1787757900.993954:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757900.993958:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757900.993960:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757900.993964:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787757900.993967:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787757900.993968:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10955 00000100:00000001:3.0:1787757900.993979:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305628865 00000100:00000001:3.0:1787757900.993989:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787757900.993993:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757900.993998:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757900.993999:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757900.994001:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757900.994011:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787757900.994013:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757900.994014:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757900.994017:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757900.994019:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757900.994022:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757900.994024:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757900.994028:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757900.994031:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757900.994041:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757900.994044:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:1.0:1787757900.994046:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757900.994052:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.994054:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757900.994058:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757900.994062:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757900.994064:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757900/real 1787757900] req@ffff8ffb4a535c00 x1874588659694208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757900.994065:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757900.994074:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757900.994078:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.994084:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757900.994088:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757900.994093:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757900.994104:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787757900.994106:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757900.994111:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:1.0:1787757900.994116:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659694592/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757900.994122:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757900.994131:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757900.994132:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757900.994136:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757900.994141:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994144:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757900.994148:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994149:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757900.994164:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659694208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757900.994176:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994181:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757900.994246:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994249:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757900.994251:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994254:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757900.994257:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757900.994261:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757900.994268:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757900.994274:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757900.994280:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659694208:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757900.994288:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994292:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757900.994296:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757900.994300:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757900.994330:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757900.994337:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757900.994342:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757900.994352:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757900.994356:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757900.994363:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757900.994366:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994371:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994382:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757900.994386:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994391:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757900.994393:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757900.994397:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757900.994399:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757900.994403:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.994408:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757900.994411:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757900.994414:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757900.994420:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694336:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757900.994425:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757900.994428:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757900.994430:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.994433:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757900.994437:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994447:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757900.994449:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:3.0:1787757900.994454:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757900.994475:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757900.994482:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757900.994485:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757900.994487:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994490:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757900.994494:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994499:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305628174 00000100:00000001:3.0:1787757900.994504:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305628865 00000100:00000001:3.0:1787757900.994507:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305628174 00000100:00000001:3.0:1787757900.994510:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305628865 00000100:00000001:3.0:1787757900.994513:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757900.994515:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994518:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757900.994523:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757900/real 1787757900] req@ffff8ffb4a537b80 x1874588659694336/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757900.994535:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757900.994536:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757900.994540:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757900.994542:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757900.994544:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994546:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757900.994548:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757900.994553:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659694336/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757900.994571:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994575:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757900.994578:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994580:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757900.994582:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994584:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757900.994599:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757900.994602:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757900.994605:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757900.994608:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757900.994612:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694336:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757900.994617:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757900.994619:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757900.994621:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.994624:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757900.994626:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757900.994629:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757900.994632:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694464:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757900.994636:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757900.994638:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757900.994640:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.994642:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757900.994646:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994652:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757900.994654:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757900.994659:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757900.994672:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757900.994677:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757900.994679:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757900.994681:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994684:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757900.994687:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994691:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305628171 00000100:00000001:3.0:1787757900.994694:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305628865 00000100:00000001:3.0:1787757900.994697:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305628171 00000100:00000001:3.0:1787757900.994700:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305628865 00000100:00000001:3.0:1787757900.994702:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757900.994704:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994707:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757900.994711:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757900/real 1787757900] req@ffff8ffb4a537800 x1874588659694464/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757900.994721:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757900.994722:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757900.994725:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757900.994727:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757900.994729:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994731:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757900.994733:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757900.994737:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659694464/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757900.994746:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994750:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757900.994755:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994758:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757900.994760:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994763:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757900.994783:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757900.994788:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757900.994796:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757900.994801:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757900.994808:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694464:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757900.994816:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757900.994819:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757900.994822:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.994826:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757900.994830:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757900.994834:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757900.994840:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757900.994847:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757900.994850:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757900.994853:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757900.994857:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757900.994861:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994869:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757900.994871:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757900.994876:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757900.994894:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757900.994900:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757900.994903:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757900.994905:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994907:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757900.994911:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994915:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305628171 00000100:00000001:3.0:1787757900.994919:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305628865 00000100:00000001:3.0:1787757900.994921:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305628171 00000100:00000001:3.0:1787757900.994924:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305628865 00000100:00000001:3.0:1787757900.994927:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757900.994929:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.994932:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757900.994937:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757900/real 1787757900] req@ffff8ffb4a534700 x1874588659694592/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757900.994955:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757900.994956:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757900.994959:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757900.994962:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757900.994963:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994965:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757900.994967:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757900.994972:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659694592/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757900.994983:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994987:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757900.994992:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.994995:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757900.994998:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.995001:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757900.995021:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757900.995027:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757900.995034:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757900.995038:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757900.995045:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757900.995053:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.995057:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757900.995060:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757900.995064:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757900.995069:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757900.995072:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757900.995075:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757900.995080:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757900.995082:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757900.995087:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757900.995089:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.995092:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757900.995093:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757900.995095:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757900.995098:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757900.995100:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757900.995101:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757900.995103:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757900.995105:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757900.995106:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757900.995108:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.995109:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757900.995111:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757900.995112:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757900.995114:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757900.995115:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757900.995117:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757900.995118:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757900.995119:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757900.995121:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757900.995123:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.995125:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757900.995130:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757900.995132:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757900.995134:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757900.995136:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757901.569328:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757901.569342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757901.569345:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757901.569354:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757901.569360:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757901.569366:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757901.569370:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757901.569374:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757901.569378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757901.761330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757901.761343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757901.761358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757901.761362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757901.761450:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757901.761458:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757901.761466:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757902.593328:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757902.593340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757902.593345:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757902.593354:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757902.593361:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757902.593367:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757902.593372:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757902.593377:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757902.593382:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757902.785372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757902.785381:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757902.785389:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757902.785391:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757902.785417:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757902.785420:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757902.785423:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757903.617337:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757903.617345:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757903.617348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757903.617354:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757903.617359:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757903.617364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757903.617368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757903.617371:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757903.617375:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757903.809343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757903.809353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757903.809365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757903.809371:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757903.809375:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757903.809381:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757903.809386:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757903.809390:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757903.809394:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757903.809399:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757903.809437:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757903.809443:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757903.809465:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757904.641311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757904.641320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757904.641323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757904.641330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757904.641335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757904.641340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757904.641343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757904.641346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757904.641350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757904.833353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757904.833362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757904.833369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757904.833371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757904.833462:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757904.833468:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757904.833473:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757905.665304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757905.665316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757905.665320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757905.665329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757905.665334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757905.665340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757905.665344:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757905.665348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757905.665352:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757905.857354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757905.857366:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757905.857370:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757905.857375:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757905.857382:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757905.857385:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757905.857391:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757905.857398:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757905.857403:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757905.857406:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757905.857507:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757905.857512:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757905.857516:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757906.113341:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757906.113358:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757906.113363:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757906.113368:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757906.113375:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757906.113385:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10960 00000100:00080000:1.0:1787757906.113403:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757906.113408:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757906.113411:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757906.113417:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757906.113421:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757906.113426:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757906.113430:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757906.113433:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757906.113434:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757906.113441:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757906.113444:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757906.113448:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757906.113453:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757906.113457:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113461:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757906.113465:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757906.113482:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757906.113484:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113490:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757906.113494:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757906.113501:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113538:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757906.113553:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659694720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757906.113570:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757906.113577:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113597:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757906.113602:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757906.113606:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757906.113609:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757906.113611:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757906.113615:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757906.113618:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10960 00000100:00080000:1.0:1787757906.113625:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757906.113630:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757906.113636:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757906.113641:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757906.113644:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757906.113649:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757906.113652:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:1.0:1787757906.113655:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757906.113662:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757906.113665:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757906.113670:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757906.113673:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757906.113676:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757906.113685:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757906.113691:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:3.0:1787757906.113695:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757906.113703:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757906.113705:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.113710:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757906.113714:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:3.0:1787757906.113724:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659694720:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757906.113732:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757906.113734:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757906.113743:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.113747:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757906.113748:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:3.0:1787757906.113757:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787757906.113760:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787757906.113763:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00000001:1.0:1787757906.113769:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113775:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.113776:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757906.113784:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757906.113788:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113792:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757906.113794:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757906.113797:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.113808:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757906.113809:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.113814:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:3.0:1787757906.113824:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:1.0:1787757906.113826:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659694848/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757906.113841:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757906.113844:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113851:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757906.113856:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757906.113861:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757906.113863:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757906.113865:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757906.113869:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757906.113874:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10960 00000100:00080000:1.0:1787757906.113881:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757906.113885:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757906.113888:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757906.113891:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757906.113893:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757906.113896:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757906.113898:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757906.113902:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757906.113906:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:3.0:1787757906.113911:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787757906.113917:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757906.113924:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757906.113928:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757906.113932:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757906.113934:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787757906.113935:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113942:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757906.113946:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757906.113947:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00000001:1.0:1787757906.113955:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.113957:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113960:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.113963:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787757906.113970:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757906.113976:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.113977:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.113987:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:3.0:1787757906.113992:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305633985 02000000:00000001:1.0:1787757906.113993:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757906.113999:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.114010:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114016:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:1.0:1787757906.114018:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659694976/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757906.114032:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305633985 00000100:00000001:1.0:1787757906.114034:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757906.114041:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.114047:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757906.114051:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787757906.114053:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787757906.114058:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757906.114064:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:3.0:1787757906.114066:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757906.114068:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757906.114080:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787757906.114081:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757906.114085:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10960 00000100:00000001:3.0:1787757906.114093:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:1.0:1787757906.114095:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757906.114102:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757906.114104:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757906.114108:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757906.114111:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757906.114115:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:3.0:1787757906.114118:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757906/real 1787757906] req@ffff8ffb4a534a80 x1874588659694720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:1.0:1787757906.114119:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757906.114126:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757906.114129:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757906.114142:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757906.114152:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757906.114160:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757906.114167:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757906.114171:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787757906.114173:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787757906.114175:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000400:00000001:3.0:1787757906.114179:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757906.114245:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757906.114250:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757906.114252:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114258:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757906.114263:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:1.0:1787757906.114269:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757906.114274:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659694720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:1.0:1787757906.114282:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.114291:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757906.114301:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757906.114307:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114310:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757906.114315:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:3.0:1787757906.114321:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:1.0:1787757906.114323:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757906.114329:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114336:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757906.114341:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114345:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757906.114354:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757906.114356:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659695104/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757906.114362:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757906.114372:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757906.114378:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00080000:3.0:1787757906.114385:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787757906.114386:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.114395:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757906.114405:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757906.114414:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757906.114426:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659694720:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757906.114436:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114452:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757906.114455:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757906.114459:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757906.114464:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757906.114468:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757906.114471:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757906.114476:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757906.114479:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757906.114483:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757906.114485:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114488:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114499:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757906.114502:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114505:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757906.114508:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757906.114510:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757906.114512:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757906.114514:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.114517:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757906.114520:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757906.114523:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757906.114528:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694848:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757906.114533:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757906.114536:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757906.114539:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.114541:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757906.114546:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114557:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757906.114559:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:3.0:1787757906.114566:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757906.114587:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757906.114594:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757906.114597:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757906.114599:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114602:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757906.114606:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114612:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305633294 00000100:00000001:3.0:1787757906.114616:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305633985 00000100:00000001:3.0:1787757906.114619:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305633294 00000100:00000001:3.0:1787757906.114623:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305633985 00000100:00000001:3.0:1787757906.114626:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757906.114628:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114631:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757906.114636:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757906/real 1787757906] req@ffff8ffb4a536300 x1874588659694848/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757906.114648:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757906.114649:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757906.114652:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757906.114654:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757906.114656:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114659:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757906.114661:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757906.114666:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659694848/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757906.114675:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114678:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757906.114681:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114683:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757906.114685:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114688:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757906.114711:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757906.114715:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757906.114719:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757906.114722:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757906.114727:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694848:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757906.114732:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757906.114734:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757906.114735:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.114738:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757906.114740:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757906.114743:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757906.114747:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694976:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757906.114751:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757906.114754:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757906.114755:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.114758:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757906.114761:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114767:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757906.114769:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757906.114774:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757906.114787:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757906.114792:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757906.114795:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757906.114797:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114799:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757906.114802:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114806:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305633291 00000100:00000001:3.0:1787757906.114809:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305633985 00000100:00000001:3.0:1787757906.114812:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305633291 00000100:00000001:3.0:1787757906.114815:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305633985 00000100:00000001:3.0:1787757906.114821:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757906.114823:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114826:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757906.114831:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757906/real 1787757906] req@ffff8ffb4a534700 x1874588659694976/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757906.114841:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757906.114843:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757906.114845:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757906.114847:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757906.114849:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114851:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757906.114853:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757906.114857:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659694976/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757906.114867:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114869:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757906.114872:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114874:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757906.114876:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.114877:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757906.114896:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757906.114899:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757906.114903:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757906.114905:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757906.114909:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659694976:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757906.114913:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757906.114915:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757906.114917:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.114920:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757906.114922:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757906.114925:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757906.114929:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757906.114933:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757906.114935:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757906.114937:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757906.114940:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757906.114943:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114948:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757906.114950:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757906.114955:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757906.114966:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757906.114971:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757906.114973:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757906.114975:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114977:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757906.114980:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.114984:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305633291 00000100:00000001:3.0:1787757906.114987:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305633985 00000100:00000001:3.0:1787757906.114991:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305633291 00000100:00000001:3.0:1787757906.114994:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305633985 00000100:00000001:3.0:1787757906.114996:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757906.114999:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.115001:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757906.115006:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757906/real 1787757906] req@ffff8ffb4a537800 x1874588659695104/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757906.115015:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757906.115016:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757906.115019:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757906.115021:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757906.115022:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115024:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757906.115026:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757906.115030:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659695104/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757906.115056:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115060:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757906.115065:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115069:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757906.115071:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115075:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757906.115100:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757906.115107:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757906.115114:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757906.115119:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757906.115126:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757906.115133:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115138:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757906.115140:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757906.115142:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757906.115146:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757906.115148:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757906.115150:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757906.115153:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757906.115154:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757906.115157:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757906.115159:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115162:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757906.115163:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757906.115165:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757906.115168:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757906.115169:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757906.115170:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757906.115173:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757906.115175:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757906.115177:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757906.115179:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115181:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757906.115183:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757906.115218:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757906.115244:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757906.115247:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757906.115249:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757906.115253:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757906.115254:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757906.115257:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757906.115259:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115262:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757906.115267:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757906.115269:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757906.115272:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757906.115275:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757906.689318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757906.689331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757906.689335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757906.689345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757906.689351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757906.689358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757906.689361:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757906.689364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757906.689368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757906.881430:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757906.881441:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757906.881450:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757906.881452:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757906.881502:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757906.881507:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757906.881511:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757907.713332:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757907.713342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757907.713345:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757907.713351:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757907.713355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757907.713361:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757907.713364:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757907.713368:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757907.713371:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757907.905307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757907.905319:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757907.905329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757907.905331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757907.905371:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757907.905376:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757907.905380:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757908.737332:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757908.737344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757908.737347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757908.737356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757908.737361:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757908.737367:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757908.737371:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757908.737375:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757908.737379:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757908.929359:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757908.929371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757908.929383:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757908.929386:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757908.929524:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757908.929531:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757908.929536:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757909.761344:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757909.761359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757909.761362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757909.761372:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757909.761378:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757909.761384:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757909.761388:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757909.761393:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757909.761397:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757909.953313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757909.953322:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757909.953329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757909.953331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757909.953428:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757909.953434:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757909.953439:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757910.785299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757910.785312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757910.785315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757910.785325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757910.785330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757910.785335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757910.785339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757910.785343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757910.785347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757910.977374:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757910.977385:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757910.977395:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757910.977397:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757910.977426:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757910.977430:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757910.977434:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757911.233318:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757911.233335:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757911.233340:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757911.233345:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757911.233352:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757911.233364:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10965 00000100:00080000:1.0:1787757911.233395:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757911.233400:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757911.233403:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757911.233410:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757911.233413:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757911.233418:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757911.233421:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757911.233425:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757911.233427:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757911.233434:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757911.233437:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757911.233441:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757911.233447:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757911.233451:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233455:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757911.233458:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757911.233477:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757911.233479:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233484:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757911.233490:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757911.233498:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233518:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757911.233533:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659695232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757911.233549:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757911.233555:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233578:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757911.233583:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757911.233588:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757911.233590:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757911.233593:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757911.233597:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757911.233600:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10965 00000100:00080000:1.0:1787757911.233605:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757911.233608:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757911.233610:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757911.233613:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757911.233616:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757911.233619:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757911.233622:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757911.233626:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757911.233629:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757911.233633:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757911.233636:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757911.233639:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757911.233648:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757911.233649:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787757911.233652:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233658:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757911.233660:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:1.0:1787757911.233665:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757911.233668:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:1.0:1787757911.233674:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.233677:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757911.233684:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233686:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.233695:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757911.233697:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757911.233703:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.233704:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757911.233707:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757911.233714:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757911.233715:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757911.233722:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757911.233730:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659695232:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757911.233733:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.233746:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:1.0:1787757911.233751:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659695360/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787757911.233757:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757911.233764:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233775:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:3.0:1787757911.233776:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757911.233784:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.233789:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233791:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757911.233801:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757911.233807:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757911.233810:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757911.233812:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757911.233816:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757911.233820:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10965 00000100:00000001:3.0:1787757911.233823:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787757911.233826:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757911.233832:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:1.0:1787757911.233833:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757911.233839:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757911.233843:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:1.0:1787757911.233845:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757911.233850:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757911.233853:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757911.233856:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757911.233860:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757911.233861:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757911.233866:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757911.233869:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757911.233871:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757911.233875:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757911.233878:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233881:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757911.233884:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757911.233887:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233890:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757911.233898:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757911.233900:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233902:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757911.233904:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757911.233910:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:3.0:1787757911.233911:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757911.233919:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757911.233931:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659695488/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757911.233934:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757911.233943:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757911.233947:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757911.233950:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.233954:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.233957:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757911.233961:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757911.233967:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787757911.233970:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757911.233973:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757911.233979:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:3.0:1787757911.233980:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305639105 00000100:00080000:1.0:1787757911.233982:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757911.233992:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757911.233995:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10965 00000100:00080000:1.0:1787757911.234001:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757911.234003:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757911.234005:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757911.234011:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757911.234014:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787757911.234017:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305639105 00000020:00000001:1.0:1787757911.234018:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757911.234024:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757911.234029:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:1.0:1787757911.234030:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757911.234041:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757911.234041:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757911.234048:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757911.234051:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757911.234060:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757911.234061:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757911.234071:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757911.234077:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757911/real 1787757911] req@ffff8ffb4a537b80 x1874588659695232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787757911.234078:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757911.234087:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757911.234098:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.234106:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757911.234112:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:3.0:1787757911.234114:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000008:00000001:1.0:1787757911.234117:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787757911.234120:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757911.234123:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757911.234155:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757911.234158:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234160:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757911.234161:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757911.234168:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787757911.234171:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:1.0:1787757911.234172:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234178:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234226:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757911.234230:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234231:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757911.234249:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659695232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757911.234258:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659695616/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757911.234274:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234287:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757911.234295:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757911.234297:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757911.234307:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234307:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757911.234312:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757911.234321:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234321:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757911.234327:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757911.234334:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757911.234341:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757911.234346:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757911.234351:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659695232:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757911.234361:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234366:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757911.234369:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757911.234372:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757911.234378:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757911.234382:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757911.234385:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757911.234392:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757911.234394:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757911.234398:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757911.234400:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234403:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234413:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757911.234416:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234419:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757911.234421:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757911.234424:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757911.234426:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757911.234428:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.234433:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757911.234437:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757911.234441:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757911.234448:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695360:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757911.234454:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757911.234458:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757911.234461:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.234465:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757911.234470:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234483:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757911.234486:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:3.0:1787757911.234496:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757911.234523:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757911.234532:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757911.234537:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757911.234540:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234544:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757911.234549:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234556:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305638414 00000100:00000001:3.0:1787757911.234560:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305639105 00000100:00000001:3.0:1787757911.234564:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305638414 00000100:00000001:3.0:1787757911.234567:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305639105 00000100:00000001:3.0:1787757911.234570:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757911.234573:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234576:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757911.234582:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757911/real 1787757911] req@ffff8ffb4a535c00 x1874588659695360/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757911.234593:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757911.234595:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757911.234598:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757911.234601:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757911.234603:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234606:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757911.234607:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757911.234612:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659695360/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757911.234622:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234625:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757911.234628:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234630:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757911.234631:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234633:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757911.234655:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757911.234658:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757911.234662:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757911.234665:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757911.234668:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695360:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757911.234673:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757911.234675:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757911.234678:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.234681:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757911.234683:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757911.234686:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757911.234689:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695488:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757911.234693:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757911.234695:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757911.234697:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.234699:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757911.234704:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234711:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757911.234712:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757911.234718:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757911.234732:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757911.234738:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757911.234740:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757911.234742:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234744:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757911.234748:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234752:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305638411 00000100:00000001:3.0:1787757911.234755:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305639105 00000100:00000001:3.0:1787757911.234758:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305638411 00000100:00000001:3.0:1787757911.234761:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305639105 00000100:00000001:3.0:1787757911.234764:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757911.234766:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234768:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757911.234773:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757911/real 1787757911] req@ffff8ffb4a537800 x1874588659695488/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757911.234782:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757911.234784:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757911.234786:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757911.234788:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757911.234789:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234792:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757911.234794:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757911.234799:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659695488/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757911.234808:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234811:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757911.234813:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234815:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757911.234817:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234819:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757911.234837:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757911.234840:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757911.234844:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757911.234846:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757911.234850:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695488:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757911.234854:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757911.234855:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757911.234857:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.234860:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757911.234862:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757911.234865:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757911.234869:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757911.234872:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757911.234874:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757911.234876:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757911.234878:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757911.234882:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234886:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757911.234888:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757911.234892:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757911.234903:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757911.234908:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757911.234910:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757911.234912:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234914:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757911.234917:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234921:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305638411 00000100:00000001:3.0:1787757911.234924:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305639105 00000100:00000001:3.0:1787757911.234927:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305638411 00000100:00000001:3.0:1787757911.234930:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305639105 00000100:00000001:3.0:1787757911.234933:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757911.234935:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.234938:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757911.234942:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757911/real 1787757911] req@ffff8ffb4a534700 x1874588659695616/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757911.234951:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757911.234952:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757911.234954:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757911.234956:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757911.234958:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234959:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757911.234961:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757911.234965:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659695616/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757911.234974:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234976:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757911.234980:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234982:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757911.234984:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.234986:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757911.235005:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757911.235008:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757911.235012:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757911.235015:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757911.235019:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757911.235024:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.235026:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757911.235028:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757911.235030:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757911.235032:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757911.235034:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757911.235036:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757911.235040:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757911.235041:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757911.235062:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757911.235064:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.235072:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757911.235074:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757911.235075:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757911.235078:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757911.235080:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757911.235081:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757911.235084:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757911.235085:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757911.235087:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757911.235088:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.235090:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757911.235091:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757911.235093:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757911.235095:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757911.235096:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757911.235097:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757911.235101:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757911.235102:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757911.235104:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757911.235105:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.235108:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757911.235112:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757911.235114:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757911.235117:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757911.235119:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757911.809311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757911.809323:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757911.809326:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757911.809334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757911.809340:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757911.809345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757911.809349:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757911.809353:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757911.809357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757912.001401:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757912.001427:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757912.001436:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757912.001438:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757912.001472:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757912.001477:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757912.001484:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757912.833323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757912.833335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757912.833339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757912.833347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757912.833353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757912.833359:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757912.833362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757912.833367:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757912.833372:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757913.025337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757913.025347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757913.025356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757913.025359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757913.025462:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757913.025467:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757913.025471:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757913.857341:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757913.857353:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757913.857356:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757913.857364:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757913.857370:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757913.857377:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757913.857380:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757913.857398:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757913.857403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757914.049267:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757914.049276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757914.049285:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757914.049290:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757914.049293:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757914.049298:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757914.049302:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757914.049306:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757914.049311:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757914.049316:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757914.049441:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757914.049447:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757914.049452:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757914.881304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757914.881313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757914.881316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757914.881323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757914.881327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757914.881332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757914.881336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757914.881339:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757914.881343:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757915.073311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757915.073319:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757915.073327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757915.073329:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757915.073353:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757915.073357:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757915.073360:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757915.905286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757915.905296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757915.905299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757915.905306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757915.905310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757915.905315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757915.905318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757915.905321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757915.905325:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757916.097332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757916.097342:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757916.097346:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757916.097351:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757916.097357:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757916.097360:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757916.097363:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757916.097366:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757916.097371:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757916.097373:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757916.097398:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757916.097401:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757916.097404:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757916.353401:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757916.353412:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757916.353416:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757916.353420:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757916.353427:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757916.353438:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10971 00000100:00080000:2.0:1787757916.353452:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757916.353457:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757916.353459:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757916.353465:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757916.353469:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757916.353473:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757916.353478:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757916.353481:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757916.353484:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757916.353490:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757916.353495:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757916.353499:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757916.353504:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757916.353509:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.353512:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757916.353516:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757916.353541:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757916.353544:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.353549:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757916.353553:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757916.353558:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.353574:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757916.353588:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659695744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757916.353603:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757916.353610:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.353634:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757916.353639:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757916.353645:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757916.353648:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757916.353651:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757916.353658:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757916.353666:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10971 00000100:00080000:2.0:1787757916.353676:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757916.353681:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757916.353686:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757916.353692:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757916.353696:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757916.353701:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757916.353707:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757916.353713:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757916.353717:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757916.353722:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787757916.353729:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757916.353731:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757916.353738:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757916.353741:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.353743:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757916.353750:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757916.353753:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757916.353759:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757916.353767:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757916.353767:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757916.353779:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757916.353780:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.353787:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:1.0:1787757916.353794:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659695744:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1787757916.353796:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757916.353806:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.353809:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757916.353812:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.353818:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787757916.353821:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757916.353824:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757916.353826:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.353832:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757916.353833:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757916.353839:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757916.353842:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.353844:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.353853:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757916.353866:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659695872/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757916.353869:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757916.353874:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:1.0:1787757916.353884:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757916.353887:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757916.353893:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.353899:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757916.353904:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757916.353908:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757916.353910:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757916.353913:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757916.353917:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757916.353922:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10971 00000100:00080000:2.0:1787757916.353928:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757916.353931:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757916.353933:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:1.0:1787757916.353939:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787757916.353941:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757916.353947:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757916.353952:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757916.353957:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787757916.353960:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757916.353961:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757916.353963:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757916.353972:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:2.0:1787757916.353973:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757916.353979:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.353981:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.353987:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787757916.353989:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757916.353996:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757916.353999:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757916.354000:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.354009:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757916.354012:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305644225 00000008:00100000:2.0:1787757916.354013:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757916.354024:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.354027:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354032:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757916.354034:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757916.354066:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354071:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305644225 00000100:00000001:2.0:1787757916.354074:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757916.354080:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757916.354084:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354085:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787757916.354094:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354099:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:2.0:1787757916.354108:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659696000/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757916.354109:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354120:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787757916.354130:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:1.0:1787757916.354134:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757916/real 1787757916] req@ffff8ffb47a38700 x1874588659695744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757916.354136:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.354145:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757916.354155:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757916.354168:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000400:00000001:1.0:1787757916.354169:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757916.354172:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000400:00000001:1.0:1787757916.354175:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:2.0:1787757916.354177:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757916.354218:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757916.354226:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787757916.354227:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757916.354233:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757916.354238:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10971 00000100:00000001:1.0:1787757916.354241:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757916.354249:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787757916.354252:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757916.354262:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:1.0:1787757916.354263:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659695744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757916.354268:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757916.354282:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757916.354287:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757916.354290:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757916.354295:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:2.0:1787757916.354298:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757916.354302:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787757916.354309:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757916.354310:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757916.354314:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757916.354322:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757916.354323:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757916.354332:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787757916.354332:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757916.354347:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757916.354357:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1787757916.354361:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757916.354370:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787757916.354374:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757916.354381:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659695744:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787757916.354382:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757916.354391:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757916.354395:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787757916.354397:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354402:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757916.354404:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757916.354408:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000008:00100000:2.0:1787757916.354411:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757916.354414:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000008:00000001:2.0:1787757916.354419:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.354424:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:2.0:1787757916.354425:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757916.354433:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757916.354437:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787757916.354438:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757916.354443:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787757916.354444:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.354450:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787757916.354452:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757916.354457:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757916.354458:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 02000000:00000001:2.0:1787757916.354463:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354464:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354472:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757916.354477:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354482:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757916.354486:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757916.354491:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659696128/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757916.354493:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757916.354498:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757916.354503:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757916.354508:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757916.354512:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.354517:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757916.354519:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787757916.354525:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.354526:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757916.354533:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757916.354536:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757916.354542:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695872:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757916.354549:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757916.354552:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757916.354555:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.354557:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757916.354563:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354572:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757916.354574:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:1.0:1787757916.354579:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757916.354599:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757916.354608:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757916.354612:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757916.354615:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354618:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757916.354624:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354630:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305643534 00000100:00000001:1.0:1787757916.354634:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305644225 00000100:00000001:1.0:1787757916.354638:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305643534 00000100:00000001:1.0:1787757916.354641:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305644225 00000100:00000001:1.0:1787757916.354645:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757916.354648:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354651:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757916.354659:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757916/real 1787757916] req@ffff8ffb47a3a300 x1874588659695872/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757916.354672:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757916.354675:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757916.354680:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757916.354684:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757916.354686:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354688:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757916.354690:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757916.354696:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659695872/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757916.354706:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354709:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757916.354712:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354714:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757916.354717:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354719:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757916.354746:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757916.354752:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757916.354757:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757916.354762:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757916.354767:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659695872:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757916.354772:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757916.354774:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757916.354775:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.354778:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757916.354780:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757916.354783:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757916.354786:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696000:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757916.354790:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757916.354792:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757916.354794:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.354796:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757916.354800:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354806:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757916.354808:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787757916.354813:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757916.354826:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757916.354831:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757916.354834:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757916.354836:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354838:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757916.354841:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354845:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305643531 00000100:00000001:1.0:1787757916.354849:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305644225 00000100:00000001:1.0:1787757916.354852:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305643531 00000100:00000001:1.0:1787757916.354854:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305644225 00000100:00000001:1.0:1787757916.354858:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757916.354860:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354863:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757916.354867:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757916/real 1787757916] req@ffff8ffb47a39180 x1874588659696000/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757916.354877:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757916.354878:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757916.354881:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757916.354882:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757916.354884:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354886:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757916.354888:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757916.354892:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659696000/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757916.354902:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354904:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757916.354907:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354909:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757916.354910:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354912:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757916.354926:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757916.354929:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757916.354932:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757916.354934:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757916.354938:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696000:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757916.354942:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354944:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757916.354946:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757916.354948:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757916.354951:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757916.354952:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757916.354955:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757916.354957:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757916.354959:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757916.354962:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757916.354964:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354966:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757916.354967:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757916.354969:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757916.354972:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757916.354974:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757916.354975:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757916.354978:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757916.354980:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757916.354982:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757916.354983:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354986:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.354989:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757916.354991:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.354993:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757916.354994:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757916.354996:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757916.354997:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757916.354999:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.355001:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757916.355003:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757916.355006:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757916.355010:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696128:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757916.355013:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757916.355015:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757916.355017:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757916.355019:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757916.355022:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.355027:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757916.355029:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787757916.355033:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757916.355066:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757916.355072:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757916.355075:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757916.355077:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.355079:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757916.355083:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.355087:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305643531 00000100:00000001:1.0:1787757916.355090:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305644225 00000100:00000001:1.0:1787757916.355093:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305643531 00000100:00000001:1.0:1787757916.355095:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305644225 00000100:00000001:1.0:1787757916.355098:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757916.355100:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.355103:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757916.355108:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757916/real 1787757916] req@ffff8ffb47a3b480 x1874588659696128/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757916.355116:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757916.355118:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757916.355120:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757916.355122:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757916.355124:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.355126:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757916.355128:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757916.355133:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659696128/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757916.355141:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.355144:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757916.355147:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.355149:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757916.355150:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.355152:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757916.355166:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757916.355169:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757916.355172:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757916.355174:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757916.355179:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696128:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757916.355210:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.355213:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757916.355216:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757916.355218:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757916.355223:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757916.355225:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757916.355227:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757916.355230:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757916.355232:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757916.355235:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757916.355238:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.355241:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.355248:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757916.355251:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.355256:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757916.355259:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757916.929232:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757916.929242:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757916.929244:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757916.929251:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757916.929255:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757916.929260:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.929263:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757916.929267:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757916.929271:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757917.121347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757917.121356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757917.121364:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757917.121366:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757917.121431:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757917.121437:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757917.121442:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757917.953286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757917.953294:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757917.953297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757917.953304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757917.953308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757917.953326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757917.953330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757917.953334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757917.953337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757918.145305:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757918.145312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757918.145320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757918.145322:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757918.145389:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757918.145396:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757918.145401:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757918.977281:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757918.977290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757918.977293:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757918.977300:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757918.977305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757918.977309:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757918.977313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757918.977316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757918.977319:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757919.169295:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757919.169304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757919.169313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757919.169316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757919.169352:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757919.169358:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757919.169364:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757920.001290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757920.001299:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757920.001302:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757920.001309:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757920.001313:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757920.001318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757920.001322:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757920.001325:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757920.001329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757920.193312:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757920.193321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757920.193329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757920.193332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757920.193366:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757920.193370:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757920.193374:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757921.025329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757921.025339:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757921.025343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757921.025349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757921.025353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757921.025358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.025361:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757921.025364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.025368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757921.217326:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757921.217336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757921.217347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757921.217348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757921.217462:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757921.217469:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757921.217474:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757921.473309:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757921.473324:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757921.473327:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757921.473332:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757921.473337:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757921.473346:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10976 00000100:00080000:2.0:1787757921.473360:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757921.473365:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757921.473367:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757921.473373:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757921.473377:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757921.473380:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757921.473384:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757921.473387:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757921.473389:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757921.473394:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757921.473398:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757921.473401:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757921.473405:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757921.473409:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473411:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757921.473414:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757921.473426:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757921.473428:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473432:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757921.473435:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757921.473441:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473455:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757921.473467:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659696256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757921.473481:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757921.473487:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473507:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757921.473512:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757921.473516:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757921.473518:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757921.473521:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757921.473524:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757921.473527:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10976 00000100:00080000:2.0:1787757921.473531:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757921.473534:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757921.473536:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757921.473539:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757921.473540:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757921.473543:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757921.473546:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757921.473548:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757921.473550:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757921.473553:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757921.473556:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757921.473558:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757921.473564:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757921.473567:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473570:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757921.473576:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757921.473581:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473584:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757921.473588:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757921.473590:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473592:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757921.473594:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757921.473597:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757921.473597:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.473608:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757921.473610:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.473615:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757921.473621:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00100000:2.0:1787757921.473623:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659696384/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757921.473625:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757921.473638:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757921.473644:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757921.473655:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473667:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:1.0:1787757921.473671:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659696256:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757921.473672:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473682:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.473685:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787757921.473689:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757921.473694:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:2.0:1787757921.473696:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787757921.473702:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473703:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757921.473710:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787757921.473712:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757921.473717:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787757921.473723:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757921.473725:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10976 00000100:00080000:2.0:1787757921.473734:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757921.473737:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757921.473739:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757921.473742:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757921.473744:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757921.473746:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757921.473749:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:2.0:1787757921.473750:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757921.473755:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:2.0:1787757921.473758:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757921.473766:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757921.473767:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787757921.473787:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757921.473792:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757921.473794:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757921.473798:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757921.473801:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473805:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757921.473808:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757921.473811:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473813:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787757921.473821:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:2.0:1787757921.473822:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757921.473829:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473833:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757921.473835:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757921.473839:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.473840:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757921.473846:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.473849:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757921.473856:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757921.473861:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659696512/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757921.473862:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757921.473873:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.473883:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305649345 00000100:00000001:2.0:1787757921.473884:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757921.473892:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473896:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757921.473900:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787757921.473903:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:2.0:1787757921.473905:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787757921.473912:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757921.473915:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757921.473916:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305649345 00000100:00000001:2.0:1787757921.473922:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757921.473926:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:2.0:1787757921.473931:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10976 00000100:00080000:2.0:1787757921.473938:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757921.473940:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757921.473942:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757921.473944:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757921.473947:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757921.473948:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787757921.473971:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787757921.473977:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757921.473985:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.473986:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757921.473991:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757921.473998:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:2.0:1787757921.473999:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757921.474008:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757921.474011:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757921.474017:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787757921.474020:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757921/real 1787757921] req@ffff8ffb47a38000 x1874588659696256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787757921.474022:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.474031:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757921.474038:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757921.474049:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.474057:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757921.474068:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:1.0:1787757921.474071:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:2.0:1787757921.474071:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787757921.474077:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757921.474079:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757921.474084:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757921.474088:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:2.0:1787757921.474088:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.474095:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474097:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757921.474105:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474112:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:2.0:1787757921.474113:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659696640/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757921.474116:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757921.474134:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:1.0:1787757921.474135:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659696256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757921.474142:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757921.474152:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474165:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474171:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757921.474175:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474179:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757921.474181:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474227:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757921.474229:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757921.474233:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757921.474239:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757921.474242:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757921.474247:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659696256:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757921.474254:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474257:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757921.474259:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757921.474262:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757921.474267:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757921.474270:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757921.474273:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757921.474278:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757921.474280:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757921.474284:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757921.474286:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474289:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474297:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757921.474299:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474302:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757921.474304:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757921.474307:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757921.474309:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757921.474311:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757921.474314:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757921.474316:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757921.474319:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757921.474323:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696384:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757921.474328:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757921.474331:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757921.474333:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757921.474335:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757921.474340:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474348:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757921.474350:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:1.0:1787757921.474355:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757921.474375:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757921.474382:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757921.474385:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757921.474387:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474389:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757921.474393:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474398:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305648654 00000100:00000001:1.0:1787757921.474402:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305649345 00000100:00000001:1.0:1787757921.474405:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305648654 00000100:00000001:1.0:1787757921.474408:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305649345 00000100:00000001:1.0:1787757921.474411:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757921.474413:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474415:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757921.474421:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757921/real 1787757921] req@ffff8ffb47a38380 x1874588659696384/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757921.474431:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757921.474433:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757921.474436:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757921.474438:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757921.474440:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474442:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757921.474444:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757921.474449:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659696384/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757921.474457:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474460:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757921.474462:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474464:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757921.474467:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474469:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757921.474483:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757921.474487:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757921.474490:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757921.474493:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757921.474497:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696384:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757921.474502:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757921.474503:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757921.474505:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757921.474507:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757921.474509:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757921.474512:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757921.474515:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696512:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757921.474519:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757921.474521:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757921.474523:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757921.474525:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757921.474529:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474534:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757921.474536:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787757921.474540:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757921.474552:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757921.474557:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757921.474560:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757921.474561:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474563:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757921.474566:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474570:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305648651 00000100:00000001:1.0:1787757921.474573:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305649345 00000100:00000001:1.0:1787757921.474576:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305648651 00000100:00000001:1.0:1787757921.474579:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305649345 00000100:00000001:1.0:1787757921.474582:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757921.474585:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474587:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757921.474591:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757921/real 1787757921] req@ffff8ffb47a38a80 x1874588659696512/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757921.474601:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757921.474602:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757921.474604:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757921.474606:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757921.474608:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474610:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757921.474612:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757921.474616:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659696512/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757921.474625:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474628:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757921.474630:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474632:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757921.474633:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474635:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757921.474648:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757921.474651:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757921.474654:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757921.474657:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757921.474660:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696512:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757921.474664:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757921.474666:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757921.474668:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757921.474670:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757921.474672:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757921.474675:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757921.474679:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696640:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757921.474683:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757921.474685:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757921.474687:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757921.474689:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757921.474692:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474696:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757921.474698:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787757921.474702:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757921.474712:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757921.474717:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757921.474719:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757921.474721:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474723:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757921.474726:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474730:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305648651 00000100:00000001:1.0:1787757921.474733:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305649345 00000100:00000001:1.0:1787757921.474736:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305648651 00000100:00000001:1.0:1787757921.474739:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305649345 00000100:00000001:1.0:1787757921.474741:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757921.474743:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474746:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757921.474750:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757921/real 1787757921] req@ffff8ffb47a3a680 x1874588659696640/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757921.474758:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757921.474760:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757921.474762:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757921.474764:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757921.474765:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474767:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757921.474769:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757921.474773:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659696640/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757921.474790:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474792:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757921.474795:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474797:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757921.474799:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474801:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757921.474812:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757921.474815:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757921.474819:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757921.474821:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757921.474825:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696640:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757921.474829:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474831:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757921.474833:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757921.474835:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757921.474837:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757921.474839:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757921.474840:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757921.474843:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757921.474844:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757921.474847:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757921.474848:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474850:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757921.474852:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757921.474854:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757921.474856:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757921.474858:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757921.474859:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757921.474861:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757921.474862:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757921.474864:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757921.474865:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474867:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757921.474868:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757921.474870:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757921.474872:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757921.474874:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757921.474876:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757921.474878:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757921.474879:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757921.474882:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757921.474884:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474886:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757921.474890:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757921.474892:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757921.474894:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757921.474896:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757922.049261:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757922.049269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757922.049271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757922.049276:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757922.049280:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757922.049284:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757922.049287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757922.049289:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757922.049291:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757922.241342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757922.241349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757922.241358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757922.241361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757922.241452:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757922.241462:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757922.241468:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757923.073336:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757923.073352:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757923.073355:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757923.073363:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757923.073369:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757923.073375:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757923.073379:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757923.073382:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757923.073386:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757923.265366:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757923.265377:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757923.265385:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757923.265391:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757923.265397:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757923.265402:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757923.265407:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757923.265410:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757923.265414:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757923.265417:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757923.265493:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757923.265502:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757923.265508:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757924.097294:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757924.097307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757924.097310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757924.097317:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757924.097322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757924.097328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757924.097331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757924.097334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757924.097338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757924.289329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757924.289342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757924.289352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757924.289355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757924.289390:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757924.289395:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757924.289400:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757925.121301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757925.121310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757925.121313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757925.121319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757925.121324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757925.121328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757925.121332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757925.121335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757925.121338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757925.313319:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757925.313329:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757925.313333:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757925.313338:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757925.313344:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757925.313347:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757925.313350:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757925.313353:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757925.313357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757925.313359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757925.313479:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757925.313485:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757925.313489:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757926.145283:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757926.145292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757926.145295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757926.145301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757926.145305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757926.145309:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757926.145312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757926.145316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757926.145319:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757926.337339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757926.337347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757926.337354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757926.337356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757926.337478:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757926.337484:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757926.337489:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757926.593352:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757926.593364:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757926.593368:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757926.593372:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757926.593378:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757926.593388:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10981 00000100:00080000:3.0:1787757926.593401:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757926.593405:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757926.593407:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757926.593413:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757926.593416:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757926.593420:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757926.593423:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757926.593426:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757926.593429:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757926.593434:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757926.593438:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757926.593442:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757926.593447:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757926.593451:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593453:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757926.593456:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757926.593468:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757926.593470:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593474:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757926.593478:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757926.593483:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593497:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757926.593510:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659696768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757926.593524:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757926.593530:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593551:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757926.593555:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757926.593560:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757926.593562:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757926.593564:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757926.593567:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757926.593571:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10981 00000100:00080000:3.0:1787757926.593576:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757926.593579:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757926.593582:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757926.593586:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757926.593588:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757926.593591:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757926.593595:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757926.593597:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757926.593600:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757926.593601:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787757926.593609:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757926.593610:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757926.593617:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757926.593618:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593624:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757926.593625:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757926.593631:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757926.593636:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757926.593640:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757926.593643:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.593646:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757926.593649:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787757926.593655:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787757926.593660:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757926.593669:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659696768:192.168.201.118@tcp:250:kworker.0 00000008:00000001:3.0:1787757926.593670:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593686:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593697:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757926.593698:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757926.593703:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.593706:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757926.593707:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757926.593713:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757926.593714:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757926.593718:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.593721:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757926.593727:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.593731:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757926.593741:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659696896/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757926.593754:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757926.593756:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757926.593758:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593764:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:3.0:1787757926.593768:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593775:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757926.593776:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757926.593783:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757926.593785:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757926.593787:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757926.593791:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757926.593795:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10981 00000100:00080000:3.0:1787757926.593801:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757926.593805:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757926.593809:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757926.593814:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757926.593817:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757926.593822:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757926.593827:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757926.593831:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757926.593834:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757926.593842:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757926.593844:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593850:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593857:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305654465 02000000:00000001:3.0:1787757926.593858:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757926.593891:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757926.593896:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593900:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757926.593905:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757926.593917:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593921:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593927:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757926.593933:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757926.593939:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593942:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305654465 00000100:00000001:3.0:1787757926.593945:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757926.593961:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787757926.593967:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757926.593971:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757926.593973:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593982:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.593990:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.593991:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594001:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757926.594009:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659697024/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757926.594015:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757926.594021:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757926.594029:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757926.594031:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757926.594034:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757926.594038:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594040:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757926.594047:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757926.594051:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594052:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757926.594063:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787757926.594065:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757926.594071:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787757926.594074:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 02000000:00000001:1.0:1787757926.594077:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.594079:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757926.594084:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757926.594088:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696896:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757926.594091:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757926.594102:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757926.594106:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787757926.594108:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10981 02000000:00000001:1.0:1787757926.594110:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.594118:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757926.594122:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787757926.594125:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.594129:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757926.594134:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757926.594138:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787757926.594141:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:3.0:1787757926.594142:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787757926.594149:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000020:00000001:3.0:1787757926.594150:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757926.594160:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757926.594163:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757926.594165:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757926.594170:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757926.594176:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757926.594179:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757926.594182:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757926.594227:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757926.594232:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.594235:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757926.594238:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757926.594241:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757926.594242:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.594252:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594256:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305653774 02000000:00000001:3.0:1787757926.594260:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757926.594264:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305654465 02000000:00000001:3.0:1787757926.594266:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.594273:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787757926.594279:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305653774 02000000:00000001:3.0:1787757926.594279:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757926.594284:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594294:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305654465 00000100:00000001:3.0:1787757926.594297:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594306:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:3.0:1787757926.594310:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659697152/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757926.594317:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594330:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757926.594331:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757926.594338:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594343:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757926.594344:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594359:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757926.594364:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757926.594370:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757926.594373:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757926.594376:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757926.594379:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757926.594382:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.594387:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757926.594392:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757926.594396:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757926.594405:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757926.594413:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757926.594416:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757926.594419:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.594423:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757926.594446:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594469:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757926.594472:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787757926.594480:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757926.594503:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594510:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305653771 00000100:00000001:1.0:1787757926.594515:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305654465 00000100:00000001:1.0:1787757926.594520:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305653771 00000100:00000001:1.0:1787757926.594524:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305654465 00000100:00000001:1.0:1787757926.594528:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757926.594533:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594537:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757926.594540:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757926.594543:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.594548:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757926.594551:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757926.594554:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757926.594561:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697152:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757926.594568:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757926.594571:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757926.594574:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757926.594577:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757926.594583:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594594:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757926.594597:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787757926.594603:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757926.594623:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594630:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305653771 00000100:00000001:1.0:1787757926.594635:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305654465 00000100:00000001:1.0:1787757926.594639:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305653771 00000100:00000001:1.0:1787757926.594643:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305654465 00000100:00000001:1.0:1787757926.594648:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757926.594652:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594657:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594661:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757926.594667:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757926.594670:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787757926.594674:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757926.594677:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757926.594680:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757926.594684:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757927.169287:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757927.169297:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757927.169299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757927.169306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757927.169311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757927.169316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757927.169320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757927.169324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757927.169328:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757927.361349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757927.361357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757927.361366:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757927.361368:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757927.361433:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757927.361440:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757927.361445:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757928.193281:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757928.193288:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757928.193291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757928.193296:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757928.193301:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757928.193305:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757928.193308:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757928.193311:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757928.193314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757928.385327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757928.385335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757928.385342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757928.385344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757928.385368:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757928.385371:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757928.385375:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757929.217340:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757929.217351:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757929.217355:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757929.217364:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757929.217370:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757929.217376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.217379:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757929.217383:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.217387:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757929.409372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757929.409380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757929.409387:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757929.409389:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757929.409494:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757929.409503:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757929.409510:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787757929.665566:0:2358:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1787757929.665624:0:2358:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787757929.665645:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757929.665662:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757929.665685:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757929.665689:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757929.665692:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757929.665703:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757929.665709:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757929.665712:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757929.665714:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757929.665716:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757929.665724:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757929.665730:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757929.665740:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757929.665742:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757929.665744:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787757929.665750:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787757929.665755:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757929.665767:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757929.665769:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757929.665773:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757929.665775:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757929.665791:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757929.665799:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757929.665845:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757926/real 1787757929] req@ffff8ffb45c33800 x1874588659697024/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757929.665879:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757929.665884:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757929.665902:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757929.665909:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757929.665912:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.665919:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757929.665923:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757929.665934:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659697024/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757929.665952:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.665960:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757929.665967:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.665971:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757929.665975:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.665980:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757929.666010:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757929.666019:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757929.666045:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757929.666051:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757929.666060:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757929.666072:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757929.666082:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757926/real 1787757929] req@ffff8ffb45c31180 x1874588659697152/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757929.666102:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757929.666105:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757929.666113:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757929.666116:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757929.666120:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666124:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757929.666126:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757929.666134:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659697152/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757929.666148:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666152:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757929.666158:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666161:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757929.666164:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666168:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757929.666239:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757929.666249:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757929.666259:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757929.666266:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757929.666275:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697152:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757929.666285:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757929.666296:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757926/real 1787757929] req@ffff8ffb45c33480 x1874588659696896/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757929.666317:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757929.666322:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757929.666332:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757929.666342:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757929.666346:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666350:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757929.666354:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757929.666364:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659696896/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757929.666379:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666383:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757929.666387:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666391:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757929.666393:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666397:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757929.666423:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757929.666428:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757929.666436:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757929.666440:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757929.666447:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659696896:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757929.666458:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757929.666468:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757926/real 1787757929] req@ffff8ffb45c30000 x1874588659696768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757929.666486:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757929.666489:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757929.666496:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757929.666499:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757929.666502:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666506:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757929.666508:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757929.666517:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659696768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757929.666532:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666536:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757929.666540:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666544:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757929.666546:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666550:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757929.666553:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757929.666558:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757929.666566:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757929.666570:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757929.666578:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659696768:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757929.666587:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666593:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757929.666597:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757929.666603:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757929.666616:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757929.666622:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757929.666627:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757929.666661:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757929.666666:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757929.666673:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757929.666678:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666683:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757929.666686:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757929.666689:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757929.666695:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757929.666698:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757929.666700:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757929.666705:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757929.666708:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757929.666712:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757929.666714:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666719:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757929.666721:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757929.666724:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757929.666729:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757929.666731:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757929.666734:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757929.666737:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757929.666740:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757929.666743:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757929.666745:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666749:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757929.666751:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757929.666755:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757929.666760:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757929.666763:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757929.666765:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757929.666768:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757929.666770:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757929.666774:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757929.666776:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666783:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757929.666798:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757929.666803:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757929.666833:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757929.666837:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757930.241334:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757930.241346:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757930.241350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757930.241359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757930.241365:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757930.241371:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757930.241375:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757930.241380:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757930.241384:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757930.433338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757930.433347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757930.433357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757930.433361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757930.433507:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757930.433515:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757930.433523:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757931.265335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757931.265349:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757931.265352:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757931.265360:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757931.265365:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757931.265371:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757931.265374:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757931.265378:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757931.265382:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757931.457377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757931.457386:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757931.457393:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757931.457395:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757931.457473:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757931.457482:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757931.457489:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757931.713376:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757931.713393:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757931.713398:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757931.713403:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757931.713411:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757931.713422:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10986 00000100:00080000:1.0:1787757931.713439:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757931.713445:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757931.713448:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757931.713453:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757931.713457:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757931.713462:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757931.713465:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757931.713468:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757931.713470:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757931.713477:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757931.713481:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.713486:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757931.713491:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757931.713496:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.713500:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757931.713505:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.713523:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757931.713528:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.713535:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757931.713541:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757931.713549:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.713574:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757931.713595:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659697280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757931.713618:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757931.713626:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.713667:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757931.713674:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757931.713682:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757931.713686:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757931.713689:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757931.713695:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757931.713702:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10986 00000100:00080000:1.0:1787757931.713711:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757931.713716:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757931.713720:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757931.713725:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757931.713729:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757931.713734:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757931.713740:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757931.713745:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757931.713749:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757931.713755:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787757931.713759:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757931.713767:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757931.713793:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.713804:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757931.713805:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757931.713814:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787757931.713816:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757931.713819:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.713828:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.713834:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757931.713835:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:3.0:1787757931.713847:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:1.0:1787757931.713850:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787757931.713859:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787757931.713863:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.713873:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757931.713877:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659697280:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757931.713887:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757931.713892:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757931.713894:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.713903:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757931.713905:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757931.713908:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757931.713915:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.713917:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.713926:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757931.713936:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.713941:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757931.713957:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659697408/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757931.713977:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757931.713978:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757931.713987:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:1.0:1787757931.713990:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.714004:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757931.714006:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757931.714017:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757931.714024:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757931.714028:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757931.714032:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757931.714040:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757931.714047:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10986 00000100:00080000:1.0:1787757931.714058:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757931.714065:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757931.714068:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757931.714073:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757931.714076:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757931.714082:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757931.714091:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000800:00000100:3.0:1787757931.714094:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757931.714101:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757931.714108:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757931.714119:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757931.714124:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757931.714126:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.714136:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757931.714137:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787757931.714144:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757931.714146:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.714159:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.714160:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757931.714170:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757931.714175:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787757931.714178:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757931.714257:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305659586 00000008:00000001:1.0:1787757931.714261:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.714271:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.714283:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757931.714287:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:1.0:1787757931.714288:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.714299:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757931.714304:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787757931.714310:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305659586 02000000:00000001:1.0:1787757931.714311:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.714332:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757931.714335:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.714345:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787757931.714356:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659697536/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757931.714357:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.714369:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757931.714387:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:3.0:1787757931.714388:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757931/real 1787757931] req@ffff8ffb4a536300 x1874588659697280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757931.714399:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.714413:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757931.714427:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000400:00000001:3.0:1787757931.714430:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757931.714438:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:1.0:1787757931.714441:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757931.714448:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757931.714453:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757931.714453:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757931.714465:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757931.714467:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787757931.714473:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757931.714481:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10986 00000100:00000001:3.0:1787757931.714484:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757931.714492:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757931.714498:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757931.714512:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:3.0:1787757931.714512:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659697280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757931.714520:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757931.714532:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757931.714539:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757931.714546:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757931.714552:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787757931.714556:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757931.714564:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757931.714566:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757931.714573:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757931.714577:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757931.714589:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757931.714597:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.714601:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757931.714608:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757931.714613:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.714618:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757931.714622:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757931.714628:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.714632:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757931.714641:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757931.714644:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.714648:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757931.714651:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757931.714655:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.714665:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757931.714676:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659697664/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757931.714693:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757931.714698:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757931.714703:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.714745:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757931.714750:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.714756:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757931.714760:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757931.714767:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757931.714804:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757931.714811:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757931.714819:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659697280:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757931.714831:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.714836:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757931.714839:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757931.714845:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757931.714853:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757931.714858:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757931.714863:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757931.714871:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757931.714875:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757931.714881:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757931.714884:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.714889:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.714904:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757931.714908:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.714913:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757931.714916:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757931.714920:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757931.714923:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757931.714927:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.714932:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757931.714936:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757931.714941:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757931.714949:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697408:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757931.714956:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757931.714961:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757931.714964:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.714968:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757931.714975:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.714990:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757931.714994:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:3.0:1787757931.715003:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757931.715036:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757931.715059:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757931.715065:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757931.715068:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715071:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757931.715077:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715086:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305658894 00000100:00000001:3.0:1787757931.715093:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305659586 00000100:00000001:3.0:1787757931.715098:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305658894 00000100:00000001:3.0:1787757931.715103:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305659586 00000100:00000001:3.0:1787757931.715107:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757931.715111:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715115:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757931.715125:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757931/real 1787757931] req@ffff8ffb4a534a80 x1874588659697408/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757931.715145:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757931.715148:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757931.715155:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757931.715159:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757931.715162:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715166:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757931.715169:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757931.715176:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659697408/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757931.715231:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715237:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757931.715242:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715246:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757931.715248:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715252:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757931.715280:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757931.715286:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757931.715294:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757931.715299:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757931.715306:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697408:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757931.715315:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757931.715318:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757931.715321:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.715326:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757931.715330:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757931.715334:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757931.715341:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757931.715348:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757931.715351:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757931.715355:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.715359:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757931.715367:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715379:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757931.715382:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757931.715402:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757931.715433:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757931.715443:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757931.715448:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757931.715451:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715454:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757931.715460:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715467:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305658892 00000100:00000001:3.0:1787757931.715471:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305659586 00000100:00000001:3.0:1787757931.715475:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305658892 00000100:00000001:3.0:1787757931.715478:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305659586 00000100:00000001:3.0:1787757931.715481:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757931.715484:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715486:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757931.715494:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757931/real 1787757931] req@ffff8ffb4a534700 x1874588659697536/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757931.715510:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757931.715513:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757931.715519:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757931.715522:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757931.715525:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715529:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757931.715532:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757931.715540:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659697536/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757931.715556:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715560:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757931.715565:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715569:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757931.715572:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715575:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757931.715600:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757931.715606:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757931.715613:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757931.715618:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757931.715623:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757931.715629:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757931.715631:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757931.715633:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.715636:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757931.715639:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757931.715642:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757931.715646:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697664:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757931.715650:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757931.715652:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757931.715654:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757931.715657:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757931.715661:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715669:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757931.715671:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757931.715677:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757931.715695:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757931.715701:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757931.715705:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757931.715707:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715709:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757931.715713:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715718:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305658892 00000100:00000001:3.0:1787757931.715721:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305659586 00000100:00000001:3.0:1787757931.715724:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305658892 00000100:00000001:3.0:1787757931.715727:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305659586 00000100:00000001:3.0:1787757931.715730:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757931.715733:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715735:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757931.715741:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757931/real 1787757931] req@ffff8ffb4a537800 x1874588659697664/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757931.715751:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757931.715753:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757931.715756:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757931.715759:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757931.715760:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715763:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757931.715765:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757931.715795:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659697664/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757931.715809:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715812:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757931.715816:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715818:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757931.715820:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715822:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757931.715842:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757931.715845:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757931.715849:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757931.715853:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757931.715857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697664:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757931.715862:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715865:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757931.715867:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757931.715870:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757931.715873:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757931.715876:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757931.715878:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757931.715883:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757931.715885:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757931.715887:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757931.715890:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715892:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757931.715894:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757931.715895:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757931.715898:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757931.715899:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757931.715901:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757931.715904:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757931.715905:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757931.715907:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757931.715908:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715910:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757931.715911:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757931.715912:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757931.715914:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757931.715915:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757931.715917:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757931.715919:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757931.715920:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757931.715922:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757931.715924:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715926:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757931.715931:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757931.715933:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757931.715935:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757931.715938:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757932.289318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757932.289336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757932.289340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757932.289351:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757932.289358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757932.289368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757932.289374:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757932.289379:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757932.289384:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757932.481376:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757932.481389:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757932.481402:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757932.481405:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757932.481439:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757932.481444:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757932.481450:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757933.313292:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757933.313302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757933.313305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757933.313311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757933.313315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757933.313319:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757933.313323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757933.313326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757933.313330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757933.505328:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757933.505336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757933.505343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757933.505348:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757933.505351:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757933.505355:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757933.505359:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757933.505362:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757933.505365:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757933.505368:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757933.505509:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757933.505515:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757933.505519:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757934.337282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757934.337292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757934.337295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757934.337303:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757934.337307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757934.337312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757934.337315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757934.337318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757934.337321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757934.529367:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757934.529378:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757934.529391:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757934.529395:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757934.529554:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757934.529563:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757934.529570:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757935.361285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757935.361295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757935.361298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757935.361304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757935.361309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757935.361313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757935.361316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757935.361320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757935.361323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757935.553370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757935.553385:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757935.553390:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757935.553394:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757935.553402:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757935.553406:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757935.553409:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757935.553412:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757935.553418:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757935.553420:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757935.553492:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757935.553501:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757935.553507:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757936.385296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757936.385306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757936.385309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757936.385316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757936.385321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757936.385326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757936.385329:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757936.385333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757936.385336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757936.577332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757936.577343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757936.577353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757936.577356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757936.577387:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757936.577392:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757936.577398:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757936.833321:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757936.833338:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757936.833342:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757936.833348:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757936.833356:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757936.833368:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10991 00000100:00080000:1.0:1787757936.833385:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757936.833392:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757936.833396:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757936.833403:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757936.833406:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757936.833410:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757936.833414:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757936.833417:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757936.833419:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757936.833425:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757936.833428:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757936.833432:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757936.833436:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757936.833440:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833443:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757936.833447:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757936.833460:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757936.833462:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833467:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757936.833470:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757936.833476:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833490:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757936.833503:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659697792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757936.833518:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757936.833523:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833546:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757936.833552:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757936.833556:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757936.833559:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757936.833561:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757936.833565:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757936.833568:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10991 00000100:00080000:1.0:1787757936.833574:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757936.833577:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757936.833579:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757936.833583:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757936.833585:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757936.833588:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757936.833592:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757936.833594:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757936.833597:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757936.833598:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787757936.833607:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:1.0:1787757936.833609:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757936.833614:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757936.833621:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.833621:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757936.833626:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757936.833631:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757936.833635:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757936.833640:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757936.833644:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757936.833645:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757936.833656:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757936.833663:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659697792:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757936.833664:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757936.833675:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757936.833679:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:1.0:1787757936.833707:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833711:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757936.833721:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757936.833725:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833733:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757936.833738:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757936.833739:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757936.833747:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757936.833749:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833757:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757936.833758:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757936.833774:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757936.833775:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659697920/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757936.833792:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757936.833796:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.833802:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757936.833803:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.833809:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00080000:1.0:1787757936.833812:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787757936.833820:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787757936.833822:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787757936.833827:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757936.833829:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757936.833833:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757936.833838:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10991 00000100:00080000:1.0:1787757936.833845:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757936.833848:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757936.833850:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757936.833854:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757936.833856:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757936.833859:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757936.833862:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757936.833865:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757936.833868:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:3.0:1787757936.833874:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787757936.833880:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757936.833886:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757936.833889:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757936.833893:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787757936.833894:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757936.833902:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.833905:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757936.833906:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757936.833914:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787757936.833915:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757936.833924:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00000001:1.0:1787757936.833925:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833931:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.833936:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757936.833939:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757936.833943:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.833945:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305664705 00000100:00000001:1.0:1787757936.833946:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757936.833951:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757936.833955:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.833963:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.833965:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:1.0:1787757936.833974:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659698048/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757936.833977:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305664705 00000100:00000001:3.0:1787757936.833990:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757936.833998:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757936.834009:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757936.834009:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834017:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757936.834021:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.834031:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757936.834032:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757936/real 1787757936] req@ffff8ffb4a535c00 x1874588659697792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757936.834043:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000400:00000001:3.0:1787757936.834059:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:1.0:1787757936.834060:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000400:00000001:3.0:1787757936.834066:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757936.834068:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:3.0:1787757936.834074:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:1.0:1787757936.834076:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757936.834080:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757936.834086:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787757936.834087:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757936.834097:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10991 00000100:00000001:3.0:1787757936.834097:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757936.834106:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757936.834109:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757936.834113:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757936.834115:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757936.834118:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757936.834120:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757936.834123:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757936.834133:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757936.834136:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757936.834140:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:3.0:1787757936.834144:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659697792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757936.834148:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757936.834157:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757936.834166:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757936.834175:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757936.834183:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834239:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:1.0:1787757936.834243:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834247:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757936.834252:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757936.834253:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757936.834260:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:1.0:1787757936.834262:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787757936.834264:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757936.834270:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00000001:1.0:1787757936.834272:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757936.834278:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787757936.834279:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757936.834293:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:1.0:1787757936.834295:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757936.834299:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:1.0:1787757936.834303:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757936.834311:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659697792:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757936.834313:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757936.834322:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787757936.834325:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757936.834331:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834332:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757936.834337:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757936.834341:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757936.834347:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:1.0:1787757936.834348:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757936.834351:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757936.834357:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757936.834363:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757936.834369:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00100000:1.0:1787757936.834372:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659698176/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757936.834373:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757936.834379:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834386:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757936.834403:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757936.834404:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757936.834408:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834412:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757936.834415:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834423:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757936.834430:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757936.834434:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757936.834437:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757936.834441:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757936.834447:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757936.834452:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757936.834458:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757936.834467:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697920:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757936.834475:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757936.834480:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757936.834483:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757936.834488:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757936.834492:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834502:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757936.834504:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:3.0:1787757936.834510:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757936.834531:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757936.834538:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757936.834541:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757936.834543:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834545:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757936.834549:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834554:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305664014 00000100:00000001:3.0:1787757936.834558:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305664705 00000100:00000001:3.0:1787757936.834561:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305664014 00000100:00000001:3.0:1787757936.834564:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305664705 00000100:00000001:3.0:1787757936.834567:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757936.834569:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834572:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757936.834578:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757936/real 1787757936] req@ffff8ffb4a537b80 x1874588659697920/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757936.834589:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757936.834590:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757936.834594:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757936.834596:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757936.834598:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834600:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757936.834602:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757936.834607:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659697920/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757936.834616:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834619:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757936.834622:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834624:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757936.834626:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834629:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757936.834645:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757936.834648:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.834655:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757936.834659:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757936.834666:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659697920:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757936.834673:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757936.834676:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757936.834679:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757936.834705:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757936.834709:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757936.834713:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757936.834720:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757936.834727:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757936.834731:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757936.834734:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757936.834738:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757936.834745:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834756:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757936.834760:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757936.834766:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757936.834788:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757936.834794:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757936.834798:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757936.834800:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834802:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757936.834806:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834810:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305664011 00000100:00000001:3.0:1787757936.834813:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305664705 00000100:00000001:3.0:1787757936.834817:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305664011 00000100:00000001:3.0:1787757936.834820:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305664705 00000100:00000001:3.0:1787757936.834823:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757936.834825:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834828:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757936.834833:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757936/real 1787757936] req@ffff8ffb4a537800 x1874588659698048/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757936.834843:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757936.834845:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757936.834848:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757936.834850:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757936.834853:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834855:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757936.834857:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757936.834862:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659698048/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757936.834870:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834873:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757936.834876:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834878:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757936.834880:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.834882:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757936.834896:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757936.834899:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.834903:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757936.834906:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757936.834910:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757936.834915:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757936.834916:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757936.834919:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757936.834922:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757936.834924:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757936.834926:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757936.834930:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698176:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757936.834934:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757936.834936:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757936.834938:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757936.834940:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757936.834943:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834949:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757936.834951:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757936.834955:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757936.834967:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757936.834972:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757936.834975:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757936.834977:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834979:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757936.834982:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.834986:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305664011 00000100:00000001:3.0:1787757936.834989:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305664705 00000100:00000001:3.0:1787757936.834992:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305664011 00000100:00000001:3.0:1787757936.834995:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305664705 00000100:00000001:3.0:1787757936.834998:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757936.835000:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.835002:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757936.835006:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757936/real 1787757936] req@ffff8ffb4a534700 x1874588659698176/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757936.835015:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757936.835017:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757936.835019:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757936.835021:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757936.835023:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835026:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757936.835028:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757936.835032:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659698176/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757936.835041:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835043:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757936.835046:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835048:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757936.835049:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835051:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757936.835065:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757936.835068:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.835071:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757936.835074:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757936.835078:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698176:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757936.835082:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835084:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757936.835086:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757936.835089:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757936.835093:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757936.835096:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757936.835098:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757936.835103:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757936.835106:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757936.835110:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757936.835113:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835116:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757936.835118:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757936.835120:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757936.835123:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757936.835126:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757936.835128:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757936.835131:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757936.835134:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757936.835137:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757936.835140:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835143:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757936.835146:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757936.835148:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757936.835152:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757936.835154:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757936.835156:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757936.835159:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757936.835162:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757936.835165:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757936.835167:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835172:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757936.835179:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757936.835182:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757936.835217:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757936.835220:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757937.409286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757937.409295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757937.409297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757937.409303:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757937.409308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757937.409312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757937.409315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757937.409319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757937.409322:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757937.601455:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757937.601467:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757937.601476:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757937.601478:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757937.601588:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757937.601595:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757937.601601:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757938.433318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757938.433327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757938.433330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757938.433336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757938.433341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757938.433346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757938.433349:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757938.433352:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757938.433356:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757938.625330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757938.625344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757938.625360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757938.625365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757938.625524:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757938.625534:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757938.625543:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757939.457290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757939.457302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757939.457305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757939.457313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757939.457319:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757939.457324:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757939.457328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757939.457332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757939.457336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757939.649302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757939.649311:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757939.649318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757939.649320:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757939.649494:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757939.649506:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757939.649516:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757940.481332:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757940.481344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757940.481347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757940.481356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757940.481362:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757940.481368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757940.481371:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757940.481375:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757940.481379:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757940.673307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757940.673316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757940.673324:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757940.673327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757940.673367:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757940.673371:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757940.673377:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757941.505296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757941.505309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757941.505313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757941.505322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757941.505327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757941.505331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757941.505335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757941.505338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757941.505342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757941.697311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757941.697323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757941.697334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757941.697336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757941.697419:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757941.697428:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757941.697435:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757941.953290:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757941.953302:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757941.953306:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757941.953310:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757941.953316:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757941.953325:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 10996 00000100:00080000:1.0:1787757941.953338:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757941.953342:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757941.953345:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757941.953351:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757941.953354:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757941.953358:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757941.953361:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757941.953364:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757941.953365:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757941.953371:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757941.953374:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757941.953377:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757941.953381:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757941.953385:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953388:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757941.953391:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757941.953405:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757941.953407:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953428:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757941.953433:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757941.953440:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953460:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757941.953477:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659698304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757941.953499:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757941.953507:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953538:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757941.953544:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757941.953551:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757941.953553:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757941.953556:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757941.953560:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757941.953564:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10996 00000100:00080000:1.0:1787757941.953570:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757941.953574:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757941.953577:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757941.953581:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757941.953583:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757941.953587:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757941.953619:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757941.953623:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757941.953625:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757941.953631:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757941.953635:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953637:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787757941.953638:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787757941.953647:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787757941.953650:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757941.953653:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787757941.953660:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953661:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757941.953665:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757941.953668:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787757941.953674:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787757941.953677:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:1.0:1787757941.953683:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.953685:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757941.953688:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.953694:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757941.953699:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757941.953705:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757941.953707:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659698304:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757941.953709:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757941.953714:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757941.953720:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953721:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757941.953731:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757941.953732:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.953735:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.953741:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:1.0:1787757941.953748:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659698432/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787757941.953751:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953766:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757941.953769:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953774:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953776:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787757941.953779:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787757941.953782:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00080000:1.0:1787757941.953792:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757941.953795:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757941.953797:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757941.953802:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757941.953805:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757941.953810:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10996 00000100:00080000:1.0:1787757941.953817:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757941.953819:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757941.953820:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757941.953823:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757941.953825:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757941.953828:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757941.953832:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787757941.953834:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757941.953836:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757941.953840:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757941.953843:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757941.953845:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757941.953848:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757941.953851:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953853:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757941.953856:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757941.953861:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:1.0:1787757941.953863:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953870:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757941.953877:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757941.953879:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953881:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757941.953883:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757941.953889:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757941.953891:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953893:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757941.953900:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953900:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953911:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:1.0:1787757941.953917:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659698560/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757941.953921:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953933:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305669825 00000100:00000001:1.0:1787757941.953938:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757941.953943:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953946:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.953950:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787757941.953951:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757941.953961:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757941.953964:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305669825 00000100:00000001:1.0:1787757941.953965:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757941.953970:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757941.953975:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787757941.953976:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757941.953983:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 10996 00000100:00000001:3.0:1787757941.953985:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757941.953989:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757941.953992:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757941.953994:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757941.954000:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757941.954001:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757941.954008:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757941.954010:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757941.954014:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757941.954017:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757941.954020:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757941.954021:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:3.0:1787757941.954026:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757941/real 1787757941] req@ffff8ffb4a534a80 x1874588659698304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757941.954028:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757941.954036:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757941.954041:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757941.954047:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757941.954059:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787757941.954062:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787757941.954063:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757941.954069:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:3.0:1787757941.954070:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000008:00000001:1.0:1787757941.954076:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954081:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757941.954082:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954090:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:1.0:1787757941.954093:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787757941.954095:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757941.954098:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954102:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757941.954104:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:3.0:1787757941.954108:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757941.954109:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757941.954116:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757941.954124:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659698304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757941.954125:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757941.954140:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659698688/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757941.954153:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954164:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757941.954165:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757941.954173:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954174:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757941.954178:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954180:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757941.954214:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954219:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757941.954222:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757941.954227:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757941.954238:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757941.954242:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757941.954248:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659698304:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757941.954269:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954275:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757941.954278:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757941.954283:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757941.954290:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757941.954294:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757941.954298:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757941.954304:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757941.954307:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757941.954312:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757941.954315:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954320:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954332:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757941.954336:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954340:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757941.954343:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757941.954346:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757941.954349:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757941.954352:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.954357:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757941.954361:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757941.954366:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757941.954374:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698432:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757941.954379:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757941.954383:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757941.954385:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.954387:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757941.954392:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954401:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757941.954403:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:3.0:1787757941.954409:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757941.954428:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757941.954435:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757941.954438:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757941.954440:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954443:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757941.954447:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954452:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305669134 00000100:00000001:3.0:1787757941.954456:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305669825 00000100:00000001:3.0:1787757941.954460:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305669134 00000100:00000001:3.0:1787757941.954463:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305669825 00000100:00000001:3.0:1787757941.954465:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757941.954468:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954470:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757941.954476:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757941/real 1787757941] req@ffff8ffb4a536300 x1874588659698432/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757941.954487:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757941.954489:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757941.954492:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757941.954494:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757941.954497:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954500:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757941.954503:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757941.954511:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659698432/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757941.954527:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954531:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757941.954536:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954540:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757941.954542:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954546:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757941.954569:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757941.954576:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757941.954583:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757941.954587:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757941.954620:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698432:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757941.954630:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757941.954632:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757941.954635:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.954639:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757941.954644:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757941.954648:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757941.954655:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757941.954661:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757941.954665:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757941.954668:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.954672:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757941.954678:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954690:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757941.954693:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787757941.954699:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757941.954720:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757941.954727:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757941.954730:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757941.954732:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954734:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757941.954738:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954743:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305669131 00000100:00000001:3.0:1787757941.954746:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305669825 00000100:00000001:3.0:1787757941.954749:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305669131 00000100:00000001:3.0:1787757941.954752:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305669825 00000100:00000001:3.0:1787757941.954755:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757941.954759:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954761:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757941.954767:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757941/real 1787757941] req@ffff8ffb4a534700 x1874588659698560/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757941.954778:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757941.954780:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757941.954783:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757941.954785:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757941.954787:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954789:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757941.954791:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757941.954796:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659698560/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757941.954805:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954808:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757941.954810:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954812:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757941.954814:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954816:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757941.954829:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757941.954832:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757941.954836:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757941.954839:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757941.954843:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757941.954848:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757941.954849:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757941.954851:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.954854:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757941.954856:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757941.954859:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757941.954862:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698688:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757941.954866:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757941.954868:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757941.954870:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757941.954873:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757941.954876:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954882:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757941.954883:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787757941.954888:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787757941.954900:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757941.954905:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757941.954908:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757941.954909:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954912:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757941.954914:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954918:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305669131 00000100:00000001:3.0:1787757941.954922:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305669825 00000100:00000001:3.0:1787757941.954924:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305669131 00000100:00000001:3.0:1787757941.954927:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305669825 00000100:00000001:3.0:1787757941.954930:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757941.954932:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.954934:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757941.954939:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757941/real 1787757941] req@ffff8ffb4a537800 x1874588659698688/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787757941.954948:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787757941.954949:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787757941.954951:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757941.954953:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757941.954955:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954957:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757941.954958:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757941.954963:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659698688/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757941.954980:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954983:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757941.954985:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954987:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757941.954989:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.954991:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757941.955002:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787757941.955005:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757941.955009:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787757941.955011:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787757941.955015:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698688:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787757941.955020:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.955022:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757941.955024:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757941.955026:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757941.955029:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757941.955031:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757941.955033:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757941.955036:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757941.955038:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757941.955041:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757941.955042:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.955045:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757941.955046:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757941.955048:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757941.955050:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757941.955051:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757941.955053:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757941.955056:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757941.955057:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757941.955059:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757941.955060:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.955062:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757941.955063:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787757941.955065:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757941.955067:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787757941.955069:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787757941.955070:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757941.955073:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787757941.955075:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757941.955077:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787757941.955079:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.955081:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757941.955086:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757941.955087:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757941.955090:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757941.955092:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757942.529298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757942.529308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757942.529311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757942.529318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757942.529322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757942.529327:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757942.529330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757942.529334:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757942.529338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757942.721371:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757942.721381:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757942.721390:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757942.721392:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757942.721478:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757942.721484:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757942.721489:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757943.553361:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757943.553373:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757943.553378:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757943.553386:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757943.553393:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757943.553400:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757943.553405:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757943.553410:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757943.553416:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757943.745359:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757943.745375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757943.745390:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757943.745397:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757943.745403:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757943.745409:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757943.745416:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757943.745421:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757943.745426:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757943.745430:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757943.745537:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757943.745548:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757943.745556:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757944.577315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757944.577341:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757944.577344:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757944.577354:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757944.577359:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757944.577366:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757944.577370:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757944.577374:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757944.577378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757944.769355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757944.769366:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757944.769378:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757944.769381:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757944.769442:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757944.769453:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757944.769462:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757945.601300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757945.601310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757945.601312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757945.601319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757945.601323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757945.601328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757945.601331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757945.601335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757945.601339:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757945.793359:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757945.793373:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757945.793378:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757945.793382:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757945.793391:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757945.793394:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757945.793397:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757945.793400:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757945.793406:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757945.793408:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757945.793475:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757945.793487:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757945.793497:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757946.625286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757946.625299:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757946.625303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757946.625310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757946.625315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757946.625320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757946.625324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757946.625328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757946.625332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757946.817368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757946.817384:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757946.817396:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757946.817398:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757946.817457:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757946.817464:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757946.817469:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757947.073354:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757947.073370:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757947.073375:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757947.073381:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757947.073388:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757947.073401:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11001 00000100:00080000:3.0:1787757947.073417:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757947.073422:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757947.073425:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757947.073432:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757947.073436:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757947.073440:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757947.073444:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757947.073447:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757947.073449:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757947.073456:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757947.073460:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757947.073464:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757947.073470:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757947.073475:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.073478:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757947.073481:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757947.073498:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757947.073521:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.073528:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757947.073532:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757947.073540:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.073561:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757947.073585:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659698816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757947.073602:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757947.073608:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.073631:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757947.073636:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757947.073642:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757947.073644:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757947.073647:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757947.073651:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757947.073655:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11001 00000100:00080000:3.0:1787757947.073662:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757947.073666:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757947.073668:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757947.073672:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757947.073674:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757947.073678:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757947.073680:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:3.0:1787757947.073682:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757947.073690:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757947.073691:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757947.073698:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787757947.073699:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757947.073706:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787757947.073709:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757947.073712:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.073718:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.073724:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757947.073725:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757947.073733:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757947.073740:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757947.073741:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757947.073752:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.073759:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:1.0:1787757947.073760:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659698816:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1787757947.073766:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787757947.073771:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:3.0:1787757947.073774:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.073779:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.073782:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787757947.073786:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757947.073788:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757947.073791:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.073797:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757947.073798:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757947.073804:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757947.073806:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.073810:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.073816:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757947.073829:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659698944/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757947.073840:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757947.073841:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757947.073849:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757947.073850:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.073863:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757947.073864:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757947.073873:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757947.073878:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757947.073881:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757947.073883:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757947.073887:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757947.073891:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11001 00000100:00080000:3.0:1787757947.073897:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757947.073899:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757947.073901:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757947.073905:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757947.073907:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757947.073911:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757947.073916:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757947.073921:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757947.073923:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757947.073932:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1787757947.073938:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757947.073947:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757947.073956:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787757947.073962:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757947.073964:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757947.073972:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787757947.073974:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.073978:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.073983:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757947.073991:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.073998:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305674945 00000100:00000001:3.0:1787757947.074001:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757947.074009:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757947.074016:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074018:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757947.074021:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074030:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305674945 02000000:00000001:3.0:1787757947.074035:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757947.074040:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074042:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757947.074045:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757947.074052:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787757947.074054:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757947.074056:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074064:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074069:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757947.074072:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757947.074082:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757947/real 1787757947] req@ffff8ffb45c32a00 x1874588659698816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757947.074091:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659699072/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787757947.074121:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757947.074124:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757947.074132:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787757947.074139:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757947.074140:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757947.074147:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074148:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074157:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757947.074158:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074161:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757947.074169:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757947.074179:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659698816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757947.074180:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757947.074183:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787757947.074232:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757947.074236:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757947.074241:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757947.074243:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787757947.074252:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757947.074254:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11001 00000100:00000001:1.0:1787757947.074259:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757947.074264:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787757947.074266:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757947.074269:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757947.074276:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757947.074277:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757947.074285:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757947.074287:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000020:00000001:3.0:1787757947.074294:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00080000:1.0:1787757947.074302:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000020:00000001:3.0:1787757947.074303:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757947.074310:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757947.074312:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:3.0:1787757947.074319:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:1.0:1787757947.074322:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659698816:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757947.074326:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757947.074334:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787757947.074337:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074345:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757947.074346:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757947.074353:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787757947.074355:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.074359:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787757947.074361:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757947.074367:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:3.0:1787757947.074369:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787757947.074376:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:3.0:1787757947.074377:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757947.074384:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787757947.074384:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757947.074394:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787757947.074396:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000008:00000001:3.0:1787757947.074398:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.074400:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787757947.074405:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074407:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757947.074413:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757947.074414:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787757947.074417:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757947.074419:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.074425:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757947.074427:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757947.074430:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074431:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757947.074437:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.074438:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074442:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757947.074449:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:3.0:1787757947.074456:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659699200/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757947.074457:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757947.074463:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757947.074468:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.074475:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757947.074480:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757947.074485:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787757947.074487:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757947.074493:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757947.074525:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698944:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757947.074529:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074534:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757947.074540:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757947.074543:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.074547:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757947.074553:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074565:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757947.074567:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787757947.074574:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757947.074594:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757947.074600:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757947.074604:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757947.074606:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074608:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757947.074612:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074618:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305674254 00000100:00000001:1.0:1787757947.074622:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305674945 00000100:00000001:1.0:1787757947.074625:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305674254 00000100:00000001:1.0:1787757947.074629:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305674945 00000100:00000001:1.0:1787757947.074632:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757947.074635:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074637:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757947.074643:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757947/real 1787757947] req@ffff8ffb45c30700 x1874588659698944/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757947.074655:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757947.074657:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757947.074660:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757947.074662:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757947.074664:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074666:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757947.074668:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757947.074673:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659698944/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757947.074682:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074685:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757947.074688:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074690:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757947.074691:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074693:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757947.074715:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757947.074718:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757947.074722:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757947.074725:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757947.074729:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659698944:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787757947.074733:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757947.074735:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757947.074737:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.074740:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757947.074743:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757947.074745:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757947.074750:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757947.074754:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757947.074756:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757947.074758:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.074761:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757947.074765:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074771:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757947.074772:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787757947.074778:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757947.074790:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757947.074794:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757947.074797:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757947.074799:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074801:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757947.074804:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074808:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305674251 00000100:00000001:1.0:1787757947.074811:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305674945 00000100:00000001:1.0:1787757947.074814:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305674251 00000100:00000001:1.0:1787757947.074818:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305674945 00000100:00000001:1.0:1787757947.074821:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757947.074823:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074825:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757947.074829:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757947/real 1787757947] req@ffff8ffb45c31500 x1874588659699072/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757947.074838:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757947.074840:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757947.074842:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757947.074844:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757947.074845:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074847:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757947.074849:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757947.074854:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659699072/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757947.074872:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074875:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757947.074878:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074880:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757947.074882:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074883:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757947.074902:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757947.074905:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757947.074909:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757947.074911:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757947.074915:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757947.074920:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074922:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757947.074924:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757947.074926:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757947.074928:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757947.074930:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757947.074932:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757947.074935:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757947.074937:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757947.074939:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757947.074941:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074943:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757947.074945:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757947.074946:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757947.074950:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757947.074952:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757947.074953:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757947.074955:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757947.074956:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757947.074958:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757947.074960:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074962:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.074966:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757947.074969:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.074971:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757947.074973:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757947.074975:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757947.074977:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757947.074979:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.074981:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787757947.074983:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757947.074986:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757947.074989:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757947.074994:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787757947.074996:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787757947.074998:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757947.075000:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787757947.075003:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.075008:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787757947.075010:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787757947.075015:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787757947.075027:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757947.075032:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757947.075035:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757947.075037:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.075039:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787757947.075042:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.075046:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305674251 00000100:00000001:1.0:1787757947.075050:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305674945 00000100:00000001:1.0:1787757947.075053:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305674251 00000100:00000001:1.0:1787757947.075056:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305674945 00000100:00000001:1.0:1787757947.075058:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757947.075060:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.075063:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757947.075067:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757947/real 1787757947] req@ffff8ffb45c30a80 x1874588659699200/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787757947.075086:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787757947.075087:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757947.075090:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787757947.075092:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787757947.075094:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.075095:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757947.075097:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787757947.075101:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659699200/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757947.075109:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.075112:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757947.075114:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.075116:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787757947.075118:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.075121:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787757947.075140:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787757947.075143:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757947.075147:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787757947.075150:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787757947.075154:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787757947.075158:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.075160:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757947.075162:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787757947.075163:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787757947.075165:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787757947.075167:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787757947.075168:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787757947.075170:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787757947.075171:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787757947.075173:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787757947.075175:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.075178:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.075208:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757947.075210:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.075213:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757947.075216:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757947.649312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757947.649324:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757947.649328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757947.649336:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757947.649342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757947.649348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.649352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757947.649356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757947.649360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757947.841329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757947.841341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757947.841355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757947.841359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757947.841439:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757947.841445:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757947.841451:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757948.673307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757948.673321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757948.673324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757948.673331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757948.673337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757948.673343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757948.673347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757948.673351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757948.673355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757948.865321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757948.865330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757948.865339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757948.865341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757948.865385:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757948.865393:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757948.865401:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757949.697298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757949.697311:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757949.697315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757949.697323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757949.697328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757949.697334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757949.697338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757949.697342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757949.697347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757949.889333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757949.889342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757949.889353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757949.889355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757949.889488:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757949.889496:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757949.889503:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757950.721305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757950.721318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757950.721321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757950.721331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757950.721337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757950.721343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757950.721346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757950.721351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757950.721355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757950.913356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757950.913365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757950.913372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757950.913374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757950.913505:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757950.913513:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757950.913520:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757951.745315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757951.745331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757951.745335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757951.745343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757951.745349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757951.745355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757951.745359:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757951.745363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757951.745367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757951.937370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757951.937379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757951.937386:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757951.937387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757951.937524:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757951.937532:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757951.937539:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757952.193385:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757952.193406:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757952.193454:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757952.193463:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757952.193474:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757952.193492:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11006 00000100:00080000:1.0:1787757952.193515:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757952.193523:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757952.193528:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757952.193536:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757952.193541:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757952.193546:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757952.193550:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757952.193553:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757952.193555:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757952.193564:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757952.193567:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757952.193571:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757952.193579:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757952.193584:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.193588:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757952.193594:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757952.193614:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757952.193617:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.193623:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757952.193628:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757952.193637:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.193658:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757952.193675:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659699328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757952.193691:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757952.193699:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.193732:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757952.193740:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757952.193750:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757952.193754:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757952.193757:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757952.193764:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757952.193771:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11006 00000100:00080000:1.0:1787757952.193782:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757952.193788:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757952.193791:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757952.193798:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:1.0:1787757952.193798:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757952.193809:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757952.193811:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:1.0:1787757952.193819:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757952.193820:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757952.193826:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000020:00000001:1.0:1787757952.193834:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757952.193835:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.193855:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757952.193858:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787757952.193859:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757952.193866:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757952.193874:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787757952.193877:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.193884:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757952.193896:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659699328:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787757952.193906:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757952.193908:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757952.193920:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757952.193925:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757952.193928:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757952.193930:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757952.193939:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757952.193942:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.193951:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757952.193961:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757952.193964:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757952.193969:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000008:00000001:1.0:1787757952.193975:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.193980:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787757952.193991:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757952.194005:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757952.194008:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.194011:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757952.194013:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757952.194019:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.194031:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757952.194049:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1787757952.194050:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659699456/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757952.194070:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757952.194071:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757952.194081:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194084:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757952.194091:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.194092:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194110:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:1.0:1787757952.194112:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757952.194122:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757952.194124:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.194126:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757952.194135:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757952.194136:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305680065 00000100:00000001:1.0:1787757952.194158:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757952.194165:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11006 00000100:00000001:0.0:1787757952.194167:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787757952.194177:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757952.194180:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305680065 00000100:00000001:1.0:1787757952.194243:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757952.194247:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757952.194254:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757952.194257:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:1.0:1787757952.194258:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757952.194265:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757952.194270:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787757952.194276:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757952.194284:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.194287:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757952.194292:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757952.194294:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757952.194310:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1787757952.194313:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757952/real 1787757952] req@ffff8ffb4a537b80 x1874588659699328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757952.194318:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757952.194334:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757952.194346:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:0.0:1787757952.194351:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:1.0:1787757952.194356:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757952.194358:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757952.194365:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757952.194373:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757952.194374:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757952.194384:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000008:00000001:1.0:1787757952.194389:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194391:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757952.194409:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194435:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757952.194442:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757952.194452:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659699328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787757952.194457:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757952.194466:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194480:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757952.194484:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757952.194488:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:1.0:1787757952.194493:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757952.194496:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.194501:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757952.194504:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787757952.194504:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194524:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757952.194528:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787757952.194531:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757952.194538:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757952.194549:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1787757952.194551:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659699584/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757952.194559:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757952.194588:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659699328:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787757952.194602:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757952.194606:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757952.194608:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194614:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757952.194618:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194619:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757952.194628:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00080000:1.0:1787757952.194629:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757952.194636:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00080000:1.0:1787757952.194637:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000020:00000001:0.0:1787757952.194648:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:1.0:1787757952.194650:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000020:00000001:0.0:1787757952.194663:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00080000:1.0:1787757952.194664:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757952.194677:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787757952.194679:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757952.194686:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00080000:1.0:1787757952.194692:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11006 00000100:00000001:0.0:1787757952.194696:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757952.194701:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787757952.194702:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757952.194707:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757952.194709:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757952.194714:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757952.194719:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757952.194720:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757952.194730:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757952.194731:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757952.194736:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757952.194741:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:1.0:1787757952.194741:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757952.194747:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757952.194751:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757952.194757:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757952.194761:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:0.0:1787757952.194763:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757952.194769:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757952.194777:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757952.194778:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.194787:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787757952.194789:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699456:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757952.194809:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757952.194812:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:1.0:1787757952.194819:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787757952.194820:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757952.194836:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787757952.194838:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757952.194845:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757952.194846:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757952.194859:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757952.194862:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787757952.194868:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194870:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:1.0:1787757952.194889:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757952.194893:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00100000:1.0:1787757952.194903:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757952.194914:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.194925:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757952.194927:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757952.194939:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757952.194944:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757952.194948:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757952.194948:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757952.194958:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787757952.194959:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194969:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.194971:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757952.194977:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305679374 02000000:00000001:1.0:1787757952.194979:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757952.194988:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305680065 02000000:00000001:1.0:1787757952.194992:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.194995:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305679374 00000100:00000001:0.0:1787757952.195001:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305680065 00000100:00000001:0.0:1787757952.195005:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757952.195005:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195023:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195031:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787757952.195032:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659699712/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757952.195045:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757952/real 1787757952] req@ffff8ffb4a535c00 x1874588659699456/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757952.195079:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757952.195092:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757952.195096:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757952.195100:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787757952.195104:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195106:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757952.195114:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757952.195120:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195125:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757952.195129:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757952.195137:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659699456/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757952.195149:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195153:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757952.195158:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195160:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757952.195163:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195165:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757952.195182:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757952.195229:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757952.195234:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757952.195237:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757952.195242:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699456:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757952.195250:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757952.195254:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757952.195259:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757952.195265:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757952.195268:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757952.195272:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757952.195277:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757952.195282:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757952.195285:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757952.195287:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757952.195289:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757952.195295:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195302:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757952.195305:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787757952.195310:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757952.195326:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757952.195333:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757952.195336:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757952.195338:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195341:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757952.195344:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195349:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305679372 00000100:00000001:0.0:1787757952.195353:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305680065 00000100:00000001:0.0:1787757952.195355:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305679372 00000100:00000001:0.0:1787757952.195358:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305680065 00000100:00000001:0.0:1787757952.195361:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757952.195364:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195366:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757952.195371:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757952/real 1787757952] req@ffff8ffb4a537800 x1874588659699584/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757952.195380:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757952.195382:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757952.195384:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757952.195386:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757952.195388:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195391:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757952.195392:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757952.195397:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659699584/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757952.195405:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195408:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757952.195438:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195440:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757952.195442:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195444:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757952.195458:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757952.195461:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757952.195465:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757952.195469:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757952.195473:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757952.195478:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195480:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757952.195482:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757952.195484:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757952.195488:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757952.195489:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757952.195491:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757952.195495:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757952.195497:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757952.195499:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757952.195501:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195504:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757952.195505:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757952.195507:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757952.195509:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757952.195510:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757952.195511:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757952.195514:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757952.195515:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757952.195517:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757952.195519:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195521:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195525:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757952.195527:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195530:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757952.195532:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757952.195533:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757952.195535:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757952.195537:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757952.195540:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757952.195542:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757952.195544:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757952.195549:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757952.195553:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757952.195555:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757952.195558:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757952.195560:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757952.195564:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195570:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757952.195571:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787757952.195576:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757952.195590:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757952.195595:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757952.195598:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757952.195600:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195602:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757952.195605:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195609:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305679372 00000100:00000001:0.0:1787757952.195613:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305680065 00000100:00000001:0.0:1787757952.195616:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305679372 00000100:00000001:0.0:1787757952.195619:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305680065 00000100:00000001:0.0:1787757952.195621:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757952.195624:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195626:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757952.195631:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757952/real 1787757952] req@ffff8ffb4a534700 x1874588659699712/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757952.195640:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757952.195641:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757952.195644:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757952.195645:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757952.195647:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195649:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757952.195651:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757952.195656:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659699712/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757952.195664:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195667:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757952.195670:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195672:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757952.195674:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195676:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757952.195691:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757952.195694:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757952.195698:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757952.195701:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757952.195705:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757952.195709:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195711:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757952.195712:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757952.195714:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757952.195716:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757952.195717:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757952.195718:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757952.195721:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757952.195722:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757952.195724:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757952.195726:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195728:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757952.195731:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757952.195733:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757952.195735:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757952.195738:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757952.769259:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757952.769267:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757952.769269:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757952.769275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757952.769278:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757952.769281:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757952.769283:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757952.769286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757952.769288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757952.961311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757952.961323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757952.961336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757952.961339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757952.961422:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757952.961438:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757952.961441:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757953.793295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757953.793307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757953.793310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757953.793319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757953.793325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757953.793331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757953.793334:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757953.793338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757953.793342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757953.985306:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757953.985315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757953.985325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757953.985332:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757953.985337:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757953.985345:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757953.985350:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757953.985354:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757953.985358:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757953.985362:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757953.985473:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757953.985480:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757953.985484:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757954.817315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757954.817328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757954.817331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757954.817339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757954.817345:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757954.817351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757954.817355:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757954.817359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757954.817390:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757955.009342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757955.009352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757955.009388:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757955.009390:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757955.009522:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757955.009528:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757955.009534:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757955.841334:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757955.841399:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757955.841403:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757955.841411:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757955.841417:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757955.841423:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757955.841427:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757955.841431:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757955.841436:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757956.033394:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757956.033408:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757956.033413:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757956.033419:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757956.033427:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757956.033432:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757956.033435:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757956.033440:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757956.033445:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757956.033447:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757956.033548:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757956.033554:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757956.033559:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757956.865367:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757956.865380:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757956.865383:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757956.865392:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757956.865399:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757956.865408:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757956.865413:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757956.865417:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757956.865423:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757957.057315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757957.057346:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757957.057357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757957.057360:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757957.057401:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757957.057409:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757957.057414:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757957.313367:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757957.313381:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757957.313384:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757957.313389:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757957.313395:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757957.313405:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11012 00000100:00080000:3.0:1787757957.313418:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757957.313422:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757957.313425:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757957.313431:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757957.313434:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757957.313437:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757957.313440:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757957.313443:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757957.313445:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757957.313452:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757957.313455:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757957.313459:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757957.313463:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757957.313468:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313470:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757957.313473:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757957.313486:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757957.313489:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313493:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757957.313496:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757957.313502:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313516:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757957.313528:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659699840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757957.313542:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757957.313548:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313574:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757957.313579:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757957.313583:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757957.313585:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757957.313588:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757957.313594:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757957.313599:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11012 00000100:00080000:3.0:1787757957.313607:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757957.313613:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757957.313616:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757957.313622:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757957.313626:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757957.313631:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757957.313637:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757957.313640:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757957.313643:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757957.313654:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757957.313655:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757957.313661:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757957.313668:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757957.313676:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757957.313676:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787757957.313694:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757957.313695:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757957.313702:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757957.313703:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757957.313717:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757957.313719:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787757957.313733:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659699840:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757957.313740:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.313744:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757957.313747:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757957.313755:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:3.0:1787757957.313756:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757957.313764:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757957.313765:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313783:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757957.313785:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787757957.313792:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757957.313796:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313798:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757957.313801:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757957.313802:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757957.313807:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313818:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757957.313830:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659699968/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757957.313842:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757957.313843:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787757957.313847:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.313854:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:3.0:1787757957.313857:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757957.313864:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757957.313870:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787757957.313872:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757957.313874:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757957.313880:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757957.313884:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757957.313888:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11012 00000100:00080000:3.0:1787757957.313895:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757957.313897:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757957.313900:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757957.313904:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757957.313908:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757957.313911:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757957.313914:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757957.313917:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757957.313919:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757957.313924:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757957.313928:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757957.313930:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757957.313933:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757957.313936:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313939:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757957.313945:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757957.313955:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:3.0:1787757957.313957:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313966:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757957.313973:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757957.313975:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.313979:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757957.313985:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757957.313987:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787757957.313997:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314001:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757957.314009:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.314009:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314029:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:3.0:1787757957.314037:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659700096/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757957.314041:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314061:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305685185 00000100:00000001:3.0:1787757957.314065:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757957.314074:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.314079:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757957.314082:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757957.314089:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787757957.314089:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757957.314103:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757957.314108:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757957.314111:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305685185 00000100:00000001:3.0:1787757957.314114:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757957.314126:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787757957.314127:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11012 00000100:00080000:3.0:1787757957.314153:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757957.314155:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757957.314178:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757957.314180:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.314241:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757957.314246:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:3.0:1787757957.314247:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757957.314253:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757957.314257:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757957.314263:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:0.0:1787757957.314272:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757957/real 1787757957] req@ffff8ffb45c30e00 x1874588659699840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757957.314273:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757957.314278:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757957.314305:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757957.314349:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757957.314352:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757957.314360:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757957.314362:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757957.314372:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787757957.314375:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757957.314379:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757957.314387:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757957.314389:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314393:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787757957.314396:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757957.314400:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:3.0:1787757957.314403:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757957.314411:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757957.314419:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659699840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757957.314419:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757957.314439:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757957.314451:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757957.314451:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757957.314476:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757957.314477:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787757957.314489:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757957.314490:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.314498:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787757957.314498:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314511:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.314518:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787757957.314520:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314523:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757957.314533:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757957.314538:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659700224/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757957.314551:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757957.314566:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787757957.314581:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:0.0:1787757957.314589:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659699840:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757957.314591:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314611:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757957.314613:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314618:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757957.314623:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757957.314628:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757957.314635:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757957.314638:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757957.314642:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757957.314649:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757957.314652:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757957.314656:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757957.314658:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.314662:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.314674:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757957.314677:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314680:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757957.314683:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757957.314685:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757957.314688:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757957.314690:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757957.314693:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757957.314696:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757957.314700:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757957.314704:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699968:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757957.314709:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757957.314713:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757957.314715:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757957.314718:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757957.314722:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314731:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757957.314733:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:0.0:1787757957.314740:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757957.314763:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757957.314770:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757957.314775:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757957.314778:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314782:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757957.314787:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314796:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305684494 00000100:00000001:0.0:1787757957.314802:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305685185 00000100:00000001:0.0:1787757957.314806:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305684494 00000100:00000001:0.0:1787757957.314811:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305685185 00000100:00000001:0.0:1787757957.314815:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757957.314820:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314824:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757957.314834:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757957/real 1787757957] req@ffff8ffb45c31c00 x1874588659699968/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757957.314854:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757957.314857:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757957.314864:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757957.314868:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757957.314871:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.314875:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757957.314877:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757957.314885:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659699968/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757957.314895:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.314899:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757957.314902:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.314904:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757957.314906:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.314908:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757957.314931:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757957.314934:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757957.314938:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757957.314942:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757957.314947:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659699968:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757957.314953:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757957.314955:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757957.314957:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757957.314961:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757957.314963:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757957.314966:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757957.314969:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757957.314974:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757957.314977:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757957.314979:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757957.314981:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757957.314986:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.314994:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757957.314996:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:0.0:1787757957.315002:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757957.315019:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757957.315026:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757957.315029:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757957.315031:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.315033:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757957.315037:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.315041:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305684491 00000100:00000001:0.0:1787757957.315045:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305685185 00000100:00000001:0.0:1787757957.315048:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305684491 00000100:00000001:0.0:1787757957.315051:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305685185 00000100:00000001:0.0:1787757957.315054:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757957.315057:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.315059:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757957.315064:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757957/real 1787757957] req@ffff8ffb45c33b80 x1874588659700096/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757957.315084:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757957.315086:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757957.315089:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757957.315091:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757957.315092:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315094:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757957.315096:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757957.315101:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659700096/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757957.315109:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315112:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757957.315115:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315117:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757957.315119:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315121:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757957.315140:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757957.315143:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757957.315146:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757957.315149:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757957.315153:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757957.315158:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757957.315161:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757957.315163:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757957.315166:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757957.315167:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757957.315170:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757957.315173:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757957.315177:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757957.315179:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757957.315182:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757957.315220:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757957.315227:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.315236:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757957.315239:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787757957.315246:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757957.315270:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757957.315279:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757957.315283:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757957.315286:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.315290:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757957.315295:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.315302:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305684492 00000100:00000001:0.0:1787757957.315307:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305685185 00000100:00000001:0.0:1787757957.315312:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305684492 00000100:00000001:0.0:1787757957.315316:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305685185 00000100:00000001:0.0:1787757957.315342:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757957.315348:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.315351:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757957.315358:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757957/real 1787757957] req@ffff8ffb45c30a80 x1874588659700224/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757957.315370:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757957.315372:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757957.315377:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757957.315379:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757957.315381:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315383:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757957.315385:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757957.315390:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659700224/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757957.315399:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315402:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757957.315406:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315408:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757957.315410:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315412:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757957.315432:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757957.315435:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757957.315440:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757957.315443:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757957.315447:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757957.315451:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315454:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757957.315455:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757957.315457:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757957.315460:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757957.315462:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757957.315464:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757957.315467:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757957.315469:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757957.315472:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757957.315474:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315476:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757957.315478:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757957.315479:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757957.315483:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757957.315484:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757957.315486:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757957.315488:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757957.315489:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757957.315492:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757957.315493:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315495:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757957.315496:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757957.315498:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757957.315499:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757957.315501:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757957.315502:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757957.315504:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757957.315506:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757957.315508:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757957.315509:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315512:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757957.315517:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757957.315519:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757957.315522:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757957.315524:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757957.889272:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757957.889284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757957.889286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757957.889294:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757957.889299:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757957.889305:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757957.889308:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757957.889322:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757957.889327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757958.081281:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757958.081291:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757958.081300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757958.081302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757958.081363:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757958.081368:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757958.081374:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757958.913255:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757958.913267:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757958.913271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757958.913280:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757958.913285:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757958.913291:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757958.913308:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757958.913313:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757958.913317:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757959.105316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757959.105327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757959.105339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757959.105343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757959.105456:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757959.105462:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757959.105466:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757959.937326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757959.937335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757959.937337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757959.937344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757959.937348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757959.937353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757959.937356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757959.937360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757959.937363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757960.129320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757960.129327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757960.129336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757960.129338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757960.129455:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757960.129464:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757960.129473:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757960.961319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757960.961329:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757960.961331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757960.961338:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757960.961343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757960.961348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757960.961352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757960.961355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757960.961359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757961.153348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757961.153368:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757961.153376:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757961.153378:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757961.153439:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757961.153445:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757961.153449:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757961.985345:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757961.985357:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757961.985361:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757961.985371:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757961.985378:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757961.985384:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757961.985389:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757961.985394:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757961.985399:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757962.177355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757962.177363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757962.177373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757962.177377:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757962.177433:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757962.177440:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757962.177444:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757962.433385:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757962.433402:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757962.433407:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757962.433412:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757962.433420:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757962.433432:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11017 00000100:00080000:2.0:1787757962.433450:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757962.433455:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757962.433459:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757962.433466:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757962.433469:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757962.433473:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757962.433477:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757962.433480:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757962.433481:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757962.433488:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757962.433491:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757962.433495:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757962.433500:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757962.433504:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433507:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757962.433510:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757962.433523:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757962.433526:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433530:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757962.433534:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757962.433539:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433557:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757962.433574:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659700352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757962.433593:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757962.433601:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433630:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757962.433637:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757962.433644:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757962.433648:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757962.433652:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757962.433657:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757962.433663:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11017 00000100:00080000:2.0:1787757962.433670:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757962.433673:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757962.433675:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757962.433678:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757962.433680:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757962.433683:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757962.433685:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757962.433688:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757962.433690:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757962.433694:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757962.433697:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757962.433699:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757962.433706:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757962.433709:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433712:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757962.433719:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757962.433723:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433726:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757962.433734:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757962.433737:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433739:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757962.433743:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757962.433743:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757962.433748:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.433753:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757962.433759:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.433760:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757962.433767:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757962.433772:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757962.433774:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659700480/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757962.433780:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757962.433787:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757962.433796:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433799:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757962.433806:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757962.433809:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659700352:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757962.433813:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757962.433824:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757962.433825:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787757962.433834:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757962.433836:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787757962.433838:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787757962.433843:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757962.433844:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757962.433854:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757962.433856:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757962.433864:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11017 02000000:00000001:0.0:1787757962.433865:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757962.433869:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757962.433875:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757962.433877:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757962.433880:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757962.433882:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757962.433884:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757962.433888:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757962.433890:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787757962.433891:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757962.433897:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:2.0:1787757962.433898:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757962.433911:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787757962.433912:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757962.433920:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757962.433922:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757962.433927:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757962.433929:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433933:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757962.433938:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757962.433941:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433943:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757962.433951:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757962.433953:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433955:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757962.433957:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757962.433959:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.433967:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757962.433969:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1787757962.433980:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659700608/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757962.433987:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757962.433995:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757962.433997:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787757962.434000:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434003:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.434007:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434009:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:2.0:1787757962.434012:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757962.434019:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757962.434020:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.434023:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757962.434030:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757962.434032:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305690305 00000100:00000001:2.0:1787757962.434035:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757962.434043:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11017 00000100:00080000:2.0:1787757962.434049:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757962.434051:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787757962.434052:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757962.434058:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757962.434063:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757962.434065:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305690305 00000020:00000001:2.0:1787757962.434067:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757962.434074:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757962.434078:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:2.0:1787757962.434080:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757962.434087:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757962.434089:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757962.434094:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757962.434098:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434101:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757962.434102:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757962.434111:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757962.434112:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757962.434120:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787757962.434122:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.434128:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757962.434132:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757962.434137:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757962/real 1787757962] req@ffff8ffb47a39500 x1874588659700352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:2.0:1787757962.434138:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.434148:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757962.434155:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757962.434157:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757962.434164:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757962.434168:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757962.434174:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757962.434258:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757962.434261:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757962.434267:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757962.434271:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757962.434274:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757962.434280:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434281:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757962.434286:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757962.434298:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659700352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757962.434307:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659700736/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757962.434326:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434337:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757962.434346:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757962.434352:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757962.434353:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757962.434360:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757962.434365:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434366:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757962.434372:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757962.434380:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:2.0:1787757962.434381:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787757962.434389:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757962.434395:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757962.434402:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659700352:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757962.434409:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434412:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757962.434414:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757962.434417:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757962.434423:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757962.434426:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757962.434429:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757962.434434:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757962.434437:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757962.434442:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757962.434443:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434446:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434454:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757962.434456:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434459:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757962.434461:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757962.434463:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757962.434465:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757962.434467:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757962.434471:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757962.434473:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757962.434476:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757962.434480:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700480:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757962.434484:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757962.434486:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757962.434488:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757962.434490:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757962.434495:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434502:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757962.434504:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:0.0:1787757962.434510:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757962.434526:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757962.434532:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757962.434535:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757962.434537:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434539:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757962.434543:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434548:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305689614 00000100:00000001:0.0:1787757962.434552:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305690305 00000100:00000001:0.0:1787757962.434555:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305689614 00000100:00000001:0.0:1787757962.434558:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305690305 00000100:00000001:0.0:1787757962.434561:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757962.434563:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434565:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757962.434570:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757962/real 1787757962] req@ffff8ffb47a39880 x1874588659700480/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757962.434580:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757962.434582:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757962.434585:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757962.434587:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757962.434588:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434591:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757962.434593:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757962.434597:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659700480/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757962.434606:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434609:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757962.434612:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434614:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757962.434616:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434618:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757962.434634:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757962.434638:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757962.434642:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757962.434645:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757962.434649:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700480:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757962.434653:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757962.434655:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757962.434657:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757962.434659:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757962.434661:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757962.434664:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757962.434667:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757962.434671:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757962.434673:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757962.434675:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757962.434678:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757962.434681:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434686:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757962.434688:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787757962.434692:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757962.434704:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757962.434708:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757962.434711:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757962.434713:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434715:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757962.434718:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434721:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305689611 00000100:00000001:0.0:1787757962.434724:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305690305 00000100:00000001:0.0:1787757962.434727:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305689611 00000100:00000001:0.0:1787757962.434730:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305690305 00000100:00000001:0.0:1787757962.434733:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757962.434735:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434738:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757962.434743:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757962/real 1787757962] req@ffff8ffb47a3a680 x1874588659700608/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757962.434752:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757962.434754:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757962.434757:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757962.434759:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757962.434760:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434762:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757962.434764:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757962.434768:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659700608/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757962.434776:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434778:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757962.434781:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434783:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757962.434785:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434787:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757962.434801:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757962.434804:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757962.434808:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757962.434811:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757962.434815:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757962.434819:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757962.434821:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757962.434822:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757962.434825:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757962.434827:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757962.434829:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757962.434833:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700736:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757962.434837:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757962.434839:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757962.434841:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757962.434843:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757962.434846:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434851:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757962.434852:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787757962.434856:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757962.434867:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757962.434872:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757962.434874:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757962.434876:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434878:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757962.434881:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434885:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305689611 00000100:00000001:0.0:1787757962.434888:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305690305 00000100:00000001:0.0:1787757962.434890:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305689611 00000100:00000001:0.0:1787757962.434894:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305690305 00000100:00000001:0.0:1787757962.434896:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757962.434898:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.434901:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757962.434905:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757962/real 1787757962] req@ffff8ffb47a38a80 x1874588659700736/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757962.434913:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757962.434915:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757962.434917:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757962.434919:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757962.434921:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434923:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757962.434925:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757962.434930:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659700736/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757962.434946:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434949:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757962.434952:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434954:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757962.434956:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434958:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757962.434971:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757962.434974:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757962.434977:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757962.434980:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757962.434983:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700736:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757962.434988:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.434990:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757962.434992:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757962.434993:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757962.434996:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757962.434998:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757962.434999:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757962.435001:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757962.435003:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757962.435005:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757962.435007:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.435009:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757962.435011:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757962.435012:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757962.435014:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757962.435015:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757962.435017:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757962.435018:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757962.435020:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757962.435023:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757962.435024:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.435026:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757962.435027:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757962.435029:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757962.435031:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757962.435033:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757962.435034:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757962.435036:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757962.435038:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757962.435041:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757962.435042:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.435045:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757962.435048:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757962.435050:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757962.435052:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757962.435054:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757963.009381:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757963.009391:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757963.009393:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757963.009400:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757963.009404:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757963.009409:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757963.009412:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757963.009415:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757963.009419:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757963.201350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757963.201357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757963.201366:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757963.201372:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757963.201375:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757963.201381:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757963.201385:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757963.201389:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757963.201391:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757963.201395:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757963.201477:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757963.201484:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757963.201489:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757964.033317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757964.033326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757964.033329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757964.033335:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757964.033340:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757964.033344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757964.033348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757964.033351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757964.033355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757964.225380:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757964.225388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757964.225396:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757964.225398:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757964.225485:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757964.225506:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757964.225511:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757965.057295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757965.057305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757965.057309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757965.057315:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757965.057320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757965.057324:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757965.057328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757965.057331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757965.057335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757965.249326:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757965.249335:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757965.249339:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757965.249345:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757965.249352:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757965.249355:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757965.249357:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757965.249360:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757965.249365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757965.249367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757965.249433:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757965.249439:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757965.249443:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757966.081315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757966.081327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757966.081330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757966.081338:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757966.081343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757966.081348:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757966.081351:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757966.081354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757966.081358:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757966.273325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757966.273332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757966.273339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757966.273341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757966.273412:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757966.273418:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757966.273423:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757967.105329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757967.105339:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757967.105342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757967.105347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757967.105353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757967.105357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757967.105360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757967.105363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757967.105366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757967.297286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757967.297295:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757967.297302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757967.297305:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757967.297442:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757967.297448:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757967.297453:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757967.553443:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757967.553454:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757967.553457:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757967.553461:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757967.553467:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757967.553476:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11022 00000100:00080000:3.0:1787757967.553489:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757967.553493:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757967.553495:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757967.553500:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757967.553503:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757967.553507:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757967.553510:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757967.553513:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757967.553515:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757967.553520:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757967.553524:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757967.553527:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757967.553531:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757967.553535:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553537:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757967.553540:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757967.553555:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757967.553557:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553560:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757967.553563:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757967.553569:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553584:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757967.553597:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659700864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757967.553611:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757967.553617:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553640:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757967.553645:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757967.553649:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757967.553651:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757967.553654:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757967.553657:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757967.553660:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11022 00000100:00080000:3.0:1787757967.553664:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757967.553667:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757967.553670:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757967.553673:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757967.553676:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757967.553679:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757967.553682:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757967.553684:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757967.553686:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757967.553690:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757967.553694:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757967.553695:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757967.553702:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757967.553703:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757967.553709:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757967.553710:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757967.553717:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757967.553718:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.553723:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553724:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757967.553734:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787757967.553735:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:3.0:1787757967.553743:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.553744:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757967.553755:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.553756:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757967.553767:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:0.0:1787757967.553771:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659700864:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787757967.553772:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553780:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757967.553785:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757967.553789:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757967.553790:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.553799:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787757967.553801:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.553806:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.553813:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787757967.553815:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659700992/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757967.553822:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553834:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757967.553838:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553843:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757967.553846:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757967.553850:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757967.553851:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757967.553853:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757967.553856:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757967.553859:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11022 00000100:00000001:0.0:1787757967.553863:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757967.553864:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757967.553870:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787757967.553871:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757967.553877:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757967.553881:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787757967.553883:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757967.553887:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757967.553890:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757967.553892:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757967.553895:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757967.553897:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757967.553901:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757967.553904:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757967.553907:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757967.553910:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757967.553914:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553917:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757967.553920:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757967.553923:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553925:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757967.553932:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000800:00000100:0.0:1787757967.553937:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787757967.553938:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553946:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757967.553949:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757967.553952:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553958:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.553959:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:3.0:1787757967.553966:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659701120/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757967.553980:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757967.553981:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757967.553988:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.553989:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554000:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757967.554001:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757967.554008:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757967.554011:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757967.554013:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757967.554020:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757967.554022:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305695425 00000100:00080000:3.0:1787757967.554023:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757967.554033:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757967.554037:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11022 00000100:00080000:3.0:1787757967.554041:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757967.554043:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757967.554044:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757967.554050:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757967.554055:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305695425 00000020:00000001:3.0:1787757967.554056:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757967.554061:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757967.554066:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757967.554070:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:3.0:1787757967.554071:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757967.554079:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757967.554082:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757967.554083:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757967.554093:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757967.554094:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757967.554101:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757967.554103:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757967.554109:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757967.554114:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787757967.554117:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757967/real 1787757967] req@ffff8ffb45c31500 x1874588659700864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787757967.554119:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.554127:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757967.554131:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757967.554167:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757967.554171:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757967.554173:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757967.554177:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757967.554221:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757967.554225:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.554228:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757967.554230:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757967.554232:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.554238:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757967.554246:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659701248/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757967.554256:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757967.554259:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757967.554262:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554266:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757967.554276:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757967.554282:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554289:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757967.554293:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757967.554305:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659700864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757967.554317:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554321:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757967.554325:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554327:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757967.554329:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554332:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757967.554335:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757967.554339:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757967.554344:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757967.554347:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757967.554352:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659700864:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757967.554358:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554361:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757967.554364:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757967.554367:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757967.554371:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757967.554374:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757967.554377:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757967.554382:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757967.554384:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757967.554387:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757967.554389:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554392:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554400:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757967.554403:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554406:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757967.554408:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757967.554410:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757967.554412:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757967.554415:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.554418:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757967.554421:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757967.554424:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757967.554428:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700992:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757967.554432:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757967.554435:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757967.554437:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.554439:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757967.554456:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554465:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757967.554468:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757967.554473:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757967.554493:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757967.554501:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757967.554504:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757967.554506:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554508:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757967.554512:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554518:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305694734 00000100:00000001:0.0:1787757967.554522:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305695425 00000100:00000001:0.0:1787757967.554525:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305694734 00000100:00000001:0.0:1787757967.554528:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305695425 00000100:00000001:0.0:1787757967.554531:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757967.554533:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554536:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757967.554541:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757967/real 1787757967] req@ffff8ffb45c30700 x1874588659700992/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757967.554554:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757967.554555:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757967.554558:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757967.554561:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757967.554562:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554564:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757967.554567:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757967.554572:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659700992/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757967.554581:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554584:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757967.554587:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554589:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757967.554590:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554592:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757967.554606:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757967.554609:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757967.554613:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757967.554616:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757967.554619:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659700992:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757967.554624:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757967.554625:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757967.554627:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.554630:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757967.554632:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757967.554635:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757967.554638:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757967.554642:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757967.554644:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757967.554647:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.554649:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757967.554652:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554658:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757967.554660:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757967.554665:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757967.554678:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757967.554683:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757967.554685:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757967.554687:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554689:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757967.554692:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554696:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305694731 00000100:00000001:0.0:1787757967.554699:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305695425 00000100:00000001:0.0:1787757967.554702:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305694731 00000100:00000001:0.0:1787757967.554705:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305695425 00000100:00000001:0.0:1787757967.554708:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757967.554710:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554712:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757967.554717:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757967/real 1787757967] req@ffff8ffb45c32a00 x1874588659701120/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757967.554727:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757967.554728:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757967.554731:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757967.554733:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757967.554734:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554737:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757967.554739:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757967.554743:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659701120/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757967.554752:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554754:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757967.554758:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554760:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757967.554761:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554764:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757967.554775:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757967.554779:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757967.554782:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757967.554785:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757967.554789:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757967.554793:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757967.554795:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757967.554797:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.554799:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757967.554801:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757967.554803:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757967.554806:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701248:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757967.554810:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757967.554812:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757967.554813:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757967.554815:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757967.554819:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554823:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757967.554825:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757967.554829:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757967.554841:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757967.554846:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757967.554848:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757967.554850:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554852:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757967.554856:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554860:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305694731 00000100:00000001:0.0:1787757967.554863:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305695425 00000100:00000001:0.0:1787757967.554866:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305694731 00000100:00000001:0.0:1787757967.554869:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305695425 00000100:00000001:0.0:1787757967.554871:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757967.554874:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.554876:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757967.554881:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757967/real 1787757967] req@ffff8ffb45c30000 x1874588659701248/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757967.554890:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757967.554891:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757967.554893:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757967.554895:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757967.554897:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554899:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757967.554900:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757967.554905:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659701248/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757967.554913:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554916:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757967.554919:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554921:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757967.554922:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554924:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757967.554937:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757967.554940:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757967.554943:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757967.554946:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757967.554950:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701248:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757967.554954:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554957:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757967.554959:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757967.554961:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757967.554964:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757967.554965:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757967.554967:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757967.554970:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757967.554971:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757967.554974:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757967.554976:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554978:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757967.554979:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757967.554981:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757967.554984:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757967.554985:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757967.554987:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757967.554988:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757967.554990:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757967.554992:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757967.554993:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.554995:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757967.554996:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757967.554998:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757967.554999:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757967.555001:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757967.555002:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757967.555003:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757967.555005:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757967.555006:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757967.555008:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.555010:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757967.555013:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757967.555015:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757967.555017:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757967.555020:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757968.129301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757968.129311:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757968.129314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757968.129321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757968.129325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757968.129330:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757968.129333:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757968.129337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757968.129340:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757968.321294:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757968.321301:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757968.321311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757968.321314:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757968.321360:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757968.321363:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757968.321368:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757969.153286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757969.153295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757969.153298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757969.153304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757969.153309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757969.153314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757969.153317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757969.153320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757969.153324:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757969.345291:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757969.345300:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757969.345307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757969.345310:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757969.345402:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757969.345408:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757969.345413:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757970.177303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757970.177312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757970.177315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757970.177323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757970.177327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757970.177332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757970.177335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757970.177338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757970.177342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757970.369295:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757970.369304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757970.369313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757970.369316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757970.369356:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757970.369362:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757970.369367:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757971.201301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757971.201310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757971.201313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757971.201319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757971.201324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757971.201328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757971.201332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757971.201335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757971.201338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757971.393346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757971.393355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757971.393362:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757971.393364:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757971.393470:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757971.393476:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757971.393481:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757972.225276:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757972.225285:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757972.225288:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757972.225295:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757972.225299:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757972.225304:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757972.225307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757972.225311:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757972.225314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757972.417320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757972.417329:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757972.417338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757972.417340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757972.417367:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757972.417370:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757972.417374:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757972.673341:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757972.673353:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787757972.673356:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757972.673360:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757972.673366:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757972.673375:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11027 00000100:00080000:1.0:1787757972.673388:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757972.673392:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757972.673395:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757972.673400:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757972.673403:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757972.673407:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787757972.673410:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787757972.673412:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757972.673414:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757972.673420:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757972.673424:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757972.673427:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757972.673431:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757972.673435:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673438:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787757972.673441:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757972.673453:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757972.673455:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673459:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757972.673463:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757972.673468:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673482:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757972.673494:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659701376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757972.673508:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787757972.673514:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673536:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757972.673540:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757972.673545:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787757972.673547:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757972.673549:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757972.673553:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757972.673556:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11027 00000100:00080000:1.0:1787757972.673560:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757972.673563:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757972.673565:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757972.673568:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757972.673570:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787757972.673572:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787757972.673575:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787757972.673577:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757972.673579:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757972.673583:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757972.673586:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757972.673588:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757972.673594:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757972.673597:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673599:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757972.673606:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757972.673610:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673614:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757972.673619:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757972.673622:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673625:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757972.673627:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757972.673629:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673635:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.673643:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:1.0:1787757972.673643:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659701504/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757972.673652:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757972.673658:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787757972.673661:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757972.673663:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757972.673666:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.673669:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673674:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757972.673679:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:1.0:1787757972.673681:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757972.673686:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787757972.673687:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787757972.673696:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673696:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757972.673703:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757972.673711:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:0.0:1787757972.673712:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659701376:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787757972.673719:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11027 00000100:00000001:0.0:1787757972.673725:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787757972.673727:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757972.673732:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757972.673734:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787757972.673735:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757972.673740:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757972.673742:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757972.673748:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787757972.673749:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:1.0:1787757972.673752:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787757972.673758:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787757972.673760:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673761:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757972.673765:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757972.673772:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787757972.673776:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757972.673778:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757972.673782:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757972.673785:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787757972.673787:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.673791:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:1.0:1787757972.673793:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757972.673800:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757972.673801:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00000001:1.0:1787757972.673808:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673812:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787757972.673819:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757972.673821:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673823:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757972.673825:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757972.673828:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673834:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757972.673840:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659701632/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787757972.673851:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000800:00000100:0.0:1787757972.673858:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787757972.673859:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673868:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787757972.673871:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757972.673876:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787757972.673878:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787757972.673880:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787757972.673883:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787757972.673887:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11027 00000100:00080000:1.0:1787757972.673891:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787757972.673894:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787757972.673896:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787757972.673899:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787757972.673901:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757972.673903:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:1.0:1787757972.673904:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787757972.673912:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787757972.673915:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757972.673918:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757972.673918:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787757972.673928:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757972.673929:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673936:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.673939:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787757972.673941:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787757972.673948:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787757972.673952:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.673955:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673955:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787757972.673959:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787757972.673965:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673968:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.673970:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305700545 02000000:00000001:1.0:1787757972.673973:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787757972.673980:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.673983:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787757972.673986:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787757972.673990:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.673992:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787757972.674002:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674005:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305700545 00000100:00000001:0.0:1787757972.674018:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00100000:1.0:1787757972.674020:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659701760/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757972.674036:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787757972.674039:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757972.674043:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757972.674070:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674077:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787757972.674080:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757972.674091:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757972/real 1787757972] req@ffff8ffb4a536300 x1874588659701376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757972.674109:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757972.674113:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757972.674120:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757972.674127:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757972.674129:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674133:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757972.674136:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757972.674143:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659701376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757972.674152:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674156:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757972.674159:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674161:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757972.674163:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674166:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757972.674169:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757972.674172:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757972.674177:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757972.674180:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757972.674221:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659701376:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757972.674228:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674231:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757972.674234:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757972.674237:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757972.674242:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757972.674245:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757972.674248:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757972.674253:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757972.674255:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757972.674259:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757972.674261:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674264:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674272:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757972.674274:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674277:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757972.674279:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757972.674281:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757972.674283:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757972.674285:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757972.674288:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757972.674291:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757972.674294:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757972.674298:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701504:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757972.674302:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757972.674305:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757972.674307:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757972.674309:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757972.674313:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674321:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757972.674322:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:0.0:1787757972.674328:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757972.674346:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757972.674353:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757972.674356:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757972.674358:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674361:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757972.674364:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674370:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305699854 00000100:00000001:0.0:1787757972.674374:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305700545 00000100:00000001:0.0:1787757972.674377:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305699854 00000100:00000001:0.0:1787757972.674380:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305700545 00000100:00000001:0.0:1787757972.674382:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757972.674385:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674387:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757972.674392:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757972/real 1787757972] req@ffff8ffb4a534a80 x1874588659701504/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757972.674402:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757972.674404:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757972.674406:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757972.674408:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757972.674410:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674412:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757972.674414:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757972.674419:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659701504/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757972.674428:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674431:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757972.674434:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674436:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757972.674438:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674440:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757972.674454:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757972.674457:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757972.674460:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757972.674463:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757972.674467:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701504:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757972.674472:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757972.674474:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757972.674475:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757972.674478:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757972.674480:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757972.674482:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757972.674486:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757972.674489:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757972.674492:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757972.674494:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757972.674496:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757972.674499:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674505:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757972.674507:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787757972.674511:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757972.674523:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757972.674528:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757972.674531:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757972.674533:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674535:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757972.674538:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674542:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305699851 00000100:00000001:0.0:1787757972.674545:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305700545 00000100:00000001:0.0:1787757972.674548:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305699851 00000100:00000001:0.0:1787757972.674551:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305700545 00000100:00000001:0.0:1787757972.674553:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757972.674555:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674558:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757972.674562:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757972/real 1787757972] req@ffff8ffb4a534700 x1874588659701632/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757972.674580:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757972.674581:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757972.674584:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757972.674586:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757972.674587:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674589:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757972.674591:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757972.674595:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659701632/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757972.674603:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674606:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757972.674608:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674610:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757972.674612:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674614:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757972.674626:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757972.674629:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757972.674632:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757972.674634:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757972.674638:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757972.674643:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757972.674644:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757972.674646:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757972.674649:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757972.674651:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757972.674653:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757972.674656:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701760:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757972.674660:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757972.674661:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757972.674664:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757972.674666:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757972.674669:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674674:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757972.674676:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787757972.674680:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757972.674691:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757972.674695:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757972.674698:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757972.674700:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674702:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757972.674705:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674708:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305699851 00000100:00000001:0.0:1787757972.674711:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305700545 00000100:00000001:0.0:1787757972.674714:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305699851 00000100:00000001:0.0:1787757972.674717:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305700545 00000100:00000001:0.0:1787757972.674720:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757972.674722:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674724:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757972.674728:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757972/real 1787757972] req@ffff8ffb4a537800 x1874588659701760/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757972.674737:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757972.674738:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757972.674741:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757972.674742:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757972.674744:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674746:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757972.674748:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757972.674753:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659701760/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757972.674761:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674763:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757972.674767:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674769:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757972.674771:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674773:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757972.674785:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757972.674788:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757972.674792:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757972.674794:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757972.674798:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659701760:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757972.674802:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674804:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757972.674806:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757972.674807:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757972.674810:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757972.674811:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757972.674813:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757972.674816:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757972.674817:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757972.674819:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757972.674821:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674823:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757972.674824:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757972.674826:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757972.674827:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757972.674829:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757972.674830:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757972.674834:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757972.674835:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757972.674837:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757972.674839:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674841:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757972.674842:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757972.674843:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757972.674845:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757972.674846:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757972.674848:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757972.674849:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757972.674851:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757972.674854:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757972.674855:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674858:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757972.674861:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757972.674864:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757972.674866:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757972.674868:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757973.249308:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757973.249317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757973.249320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757973.249326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757973.249331:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757973.249336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757973.249339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757973.249342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757973.249345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757973.441337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757973.441344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757973.441355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757973.441360:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757973.441363:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757973.441367:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757973.441372:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757973.441375:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757973.441377:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757973.441380:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757973.441448:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757973.441454:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757973.441458:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757974.273342:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757974.273355:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757974.273358:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757974.273366:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757974.273371:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757974.273377:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757974.273380:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757974.273385:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757974.273390:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757974.465327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757974.465336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757974.465344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757974.465346:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757974.465435:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757974.465445:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757974.465466:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757975.297313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757975.297321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757975.297324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757975.297330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757975.297335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757975.297340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757975.297343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757975.297346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757975.297350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757975.489336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757975.489346:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757975.489349:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757975.489353:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757975.489359:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757975.489362:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757975.489365:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757975.489368:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757975.489373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757975.489375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757975.489486:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757975.489494:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757975.489501:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757976.321271:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757976.321281:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757976.321284:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757976.321291:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757976.321295:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757976.321300:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757976.321303:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757976.321306:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757976.321310:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757976.513299:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757976.513307:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757976.513314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757976.513316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757976.513345:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757976.513348:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757976.513352:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757977.345274:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757977.345283:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757977.345286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757977.345293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757977.345298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757977.345303:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757977.345306:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757977.345310:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757977.345313:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757977.537307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757977.537316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757977.537323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757977.537324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757977.537413:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757977.537419:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757977.537424:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757977.793332:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757977.793344:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757977.793347:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757977.793352:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757977.793358:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757977.793367:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11032 00000100:00080000:3.0:1787757977.793381:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757977.793385:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757977.793388:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757977.793393:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757977.793396:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757977.793400:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757977.793403:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757977.793406:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757977.793408:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757977.793413:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757977.793417:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757977.793420:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757977.793424:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757977.793428:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793430:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757977.793433:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757977.793446:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757977.793449:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793453:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757977.793456:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757977.793461:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793475:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757977.793487:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659701888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757977.793501:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757977.793507:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793529:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757977.793534:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757977.793539:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757977.793541:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757977.793543:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757977.793547:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757977.793550:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11032 00000100:00080000:3.0:1787757977.793554:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757977.793556:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757977.793558:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757977.793561:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757977.793563:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757977.793566:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757977.793569:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757977.793571:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757977.793573:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757977.793576:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757977.793579:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757977.793581:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757977.793587:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757977.793590:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793592:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757977.793598:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757977.793602:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793605:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757977.793610:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757977.793612:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793615:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757977.793617:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757977.793620:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793625:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.793637:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757977.793646:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:3.0:1787757977.793648:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659702016/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757977.793651:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757977.793657:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757977.793663:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757977.793672:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757977.793681:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757977.793687:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757977.793693:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793694:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793707:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757977.793716:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659701888:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787757977.793721:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757977.793735:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787757977.793740:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757977.793745:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787757977.793746:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787757977.793748:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787757977.793757:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793758:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757977.793767:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11032 02000000:00000001:0.0:1787757977.793770:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757977.793774:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757977.793779:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757977.793782:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787757977.793783:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757977.793786:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757977.793790:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757977.793795:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757977.793798:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757977.793800:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757977.793802:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757977.793807:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757977.793810:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757977.793812:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757977.793816:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757977.793817:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787757977.793819:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.793822:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:3.0:1787757977.793825:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757977.793832:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757977.793834:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00000001:3.0:1787757977.793838:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793843:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757977.793850:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757977.793851:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793862:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757977.793864:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757977.793867:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793873:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757977.793880:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659702144/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757977.793894:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757977.793902:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757977.793907:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793911:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757977.793914:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757977.793915:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787757977.793923:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757977.793927:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757977.793928:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757977.793931:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757977.793937:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.793938:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757977.793945:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11032 00000100:00080000:3.0:1787757977.793950:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757977.793952:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757977.793954:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757977.793956:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757977.793958:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757977.793981:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757977.793988:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757977.793989:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757977.793992:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757977.793997:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757977.794003:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757977.794006:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.794007:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757977.794012:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757977.794016:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757977.794018:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.794026:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757977.794029:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757977.794032:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305705665 00000008:00000001:3.0:1787757977.794034:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.794043:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757977.794050:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757977.794053:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.794055:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757977.794058:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787757977.794059:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757977.794065:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794073:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305705665 00000100:00000001:3.0:1787757977.794074:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757977.794089:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659702272/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757977.794091:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757977.794101:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757977.794110:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757977.794111:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757977.794116:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794120:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787757977.794123:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757977.794134:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757977/real 1787757977] req@ffff8ffb45c33480 x1874588659701888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757977.794153:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757977.794157:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757977.794162:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757977.794166:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757977.794169:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794172:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757977.794175:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757977.794226:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659701888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757977.794238:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794243:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757977.794247:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794249:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757977.794250:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794253:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757977.794256:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757977.794259:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757977.794264:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757977.794268:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757977.794273:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659701888:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757977.794279:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794282:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757977.794284:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757977.794288:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757977.794292:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757977.794295:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757977.794298:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757977.794303:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757977.794305:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757977.794308:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757977.794310:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794313:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794322:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757977.794324:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794327:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757977.794329:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757977.794331:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757977.794333:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757977.794335:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757977.794338:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757977.794341:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757977.794344:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757977.794348:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702016:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757977.794352:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757977.794355:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757977.794357:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757977.794360:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757977.794364:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794372:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757977.794374:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787757977.794379:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757977.794399:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757977.794405:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757977.794408:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757977.794410:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794412:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757977.794416:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794422:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305704974 00000100:00000001:0.0:1787757977.794425:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305705665 00000100:00000001:0.0:1787757977.794428:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305704974 00000100:00000001:0.0:1787757977.794431:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305705665 00000100:00000001:0.0:1787757977.794434:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757977.794437:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794439:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757977.794444:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757977/real 1787757977] req@ffff8ffb45c31180 x1874588659702016/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757977.794464:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757977.794465:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757977.794469:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757977.794471:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757977.794472:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794475:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757977.794477:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757977.794482:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659702016/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757977.794490:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794493:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757977.794496:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794498:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757977.794499:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794501:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757977.794516:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757977.794519:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757977.794523:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757977.794526:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757977.794530:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702016:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757977.794534:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757977.794536:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757977.794538:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757977.794540:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757977.794542:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757977.794545:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757977.794548:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757977.794551:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757977.794553:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757977.794556:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757977.794558:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757977.794561:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794567:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757977.794569:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787757977.794574:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757977.794586:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757977.794591:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757977.794593:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757977.794595:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794597:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757977.794600:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794604:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305704971 00000100:00000001:0.0:1787757977.794607:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305705665 00000100:00000001:0.0:1787757977.794610:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305704971 00000100:00000001:0.0:1787757977.794613:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305705665 00000100:00000001:0.0:1787757977.794616:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757977.794618:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794620:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757977.794625:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757977/real 1787757977] req@ffff8ffb45c33800 x1874588659702144/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757977.794634:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757977.794635:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757977.794637:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757977.794639:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757977.794641:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794643:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757977.794645:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757977.794650:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659702144/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757977.794658:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794661:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757977.794664:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794666:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757977.794668:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794670:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757977.794686:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757977.794692:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757977.794698:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757977.794702:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757977.794709:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757977.794715:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757977.794718:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757977.794721:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757977.794725:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757977.794728:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757977.794733:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757977.794739:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702272:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757977.794746:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757977.794749:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757977.794752:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757977.794756:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757977.794762:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794772:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757977.794775:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757977.794781:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757977.794802:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757977.794808:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757977.794811:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757977.794813:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794815:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757977.794819:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794823:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305704971 00000100:00000001:0.0:1787757977.794827:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305705665 00000100:00000001:0.0:1787757977.794830:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305704971 00000100:00000001:0.0:1787757977.794833:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305705665 00000100:00000001:0.0:1787757977.794836:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757977.794838:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.794841:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757977.794846:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757977/real 1787757977] req@ffff8ffb45c30000 x1874588659702272/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757977.794855:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757977.794857:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757977.794860:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757977.794862:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757977.794864:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794866:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757977.794868:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757977.794872:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659702272/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757977.794881:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794884:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757977.794887:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794889:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757977.794890:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794893:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757977.794905:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757977.794908:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757977.794911:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757977.794914:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757977.794918:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702272:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757977.794922:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794925:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757977.794927:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757977.794929:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757977.794932:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757977.794935:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757977.794938:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757977.794942:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757977.794945:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757977.794949:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757977.794951:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.794955:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757977.794958:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757977.794986:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757977.794992:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757977.794994:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757977.794997:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757977.795000:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757977.795003:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757977.795006:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757977.795009:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.795013:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757977.795016:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757977.795018:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757977.795021:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757977.795023:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757977.795025:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757977.795029:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757977.795031:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757977.795035:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757977.795037:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.795041:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757977.795048:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757977.795051:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757977.795054:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757977.795057:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757978.369286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757978.369295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757978.369298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757978.369304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757978.369308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757978.369313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757978.369316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757978.369320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757978.369324:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757978.561346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757978.561359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757978.561372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757978.561375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757978.561447:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757978.561453:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757978.561457:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757979.393316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757979.393328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757979.393332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757979.393342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757979.393349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757979.393356:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757979.393377:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757979.393381:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757979.393385:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757979.585320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757979.585329:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757979.585336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757979.585338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757979.585432:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757979.585438:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757979.585443:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757980.417280:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757980.417290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757980.417293:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757980.417299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757980.417304:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757980.417320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757980.417323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757980.417327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757980.417331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757980.609336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757980.609347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757980.609357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757980.609359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757980.609396:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757980.609401:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757980.609407:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757981.441307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757981.441316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757981.441319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757981.441325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757981.441330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757981.441334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757981.441338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757981.441341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757981.441344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757981.633347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757981.633355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757981.633362:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757981.633363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757981.633458:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757981.633464:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757981.633469:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757982.465312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757982.465320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757982.465323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757982.465329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757982.465334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757982.465339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757982.465343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757982.465347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757982.465350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757982.657310:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757982.657318:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757982.657342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757982.657345:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757982.657425:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757982.657431:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757982.657437:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757982.913272:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757982.913288:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757982.913293:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757982.913299:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757982.913306:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757982.913318:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11037 00000100:00080000:3.0:1787757982.913334:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757982.913340:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757982.913345:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757982.913353:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757982.913357:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757982.913362:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757982.913368:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757982.913372:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757982.913376:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757982.913383:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757982.913387:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757982.913391:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757982.913395:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757982.913400:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913403:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757982.913407:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757982.913418:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757982.913421:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913425:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757982.913429:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757982.913434:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913448:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757982.913460:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659702400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757982.913475:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757982.913495:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913516:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757982.913521:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757982.913526:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757982.913528:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757982.913531:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757982.913534:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757982.913537:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11037 00000100:00080000:3.0:1787757982.913543:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757982.913545:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757982.913547:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757982.913550:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757982.913552:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757982.913555:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757982.913557:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757982.913561:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757982.913564:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757982.913568:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757982.913573:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757982.913576:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757982.913582:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757982.913586:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913588:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757982.913594:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757982.913600:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00000001:3.0:1787757982.913600:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913609:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.913610:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757982.913627:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757982.913628:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787757982.913635:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.913636:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757982.913640:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.913646:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757982.913652:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757982.913654:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757982.913657:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757982.913662:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757982.913665:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757982.913676:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659702400:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757982.913677:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.913689:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:3.0:1787757982.913695:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659702528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757982.913698:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757982.913706:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.913715:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757982.913721:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757982.913725:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913727:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913739:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757982.913744:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757982.913749:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757982.913751:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757982.913753:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757982.913755:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757982.913761:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:3.0:1787757982.913764:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757982.913771:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787757982.913773:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11037 00000100:00080000:3.0:1787757982.913782:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757982.913785:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757982.913786:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757982.913791:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757982.913793:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757982.913796:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757982.913799:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757982.913802:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757982.913804:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757982.913818:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1787757982.913826:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787757982.913828:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757982.913835:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757982.913840:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757982.913844:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.913846:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757982.913848:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757982.913855:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757982.913856:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757982.913863:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787757982.913864:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.913899:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787757982.913899:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757982.913916:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757982.913919:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757982.913921:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.913928:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757982.913929:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305710785 02000000:00000001:3.0:1787757982.913931:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757982.913937:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.913949:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757982.913950:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.913960:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305710785 00000100:00100000:3.0:1787757982.913972:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659702656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757982.913973:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757982.913981:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757982.913990:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.913998:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757982.914013:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757982/real 1787757982] req@ffff8ffb45c32a00 x1874588659702400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757982.914013:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757982.914024:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.914037:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757982.914040:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757982.914047:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:3.0:1787757982.914047:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757982.914054:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:3.0:1787757982.914060:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757982.914061:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757982.914066:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757982.914068:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787757982.914071:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787757982.914074:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757982.914077:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757982.914084:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:0.0:1787757982.914088:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659702400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757982.914096:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11037 00000100:00080000:3.0:1787757982.914111:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757982.914113:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757982.914120:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757982.914123:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757982.914125:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757982.914132:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787757982.914134:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757982.914137:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:3.0:1787757982.914140:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757982.914142:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914151:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000020:00000001:3.0:1787757982.914152:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757982.914155:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787757982.914165:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:0.0:1787757982.914166:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787757982.914178:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00080000:0.0:1787757982.914179:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757982.914225:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757982.914231:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659702400:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757982.914238:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914245:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787757982.914247:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757982.914250:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757982.914258:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00080000:3.0:1787757982.914264:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787757982.914267:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787757982.914271:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757982.914273:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:3.0:1787757982.914281:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757982.914290:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:0.0:1787757982.914294:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787757982.914298:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.914305:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787757982.914307:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787757982.914315:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000008:00100000:3.0:1787757982.914318:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757982.914325:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000008:00000001:3.0:1787757982.914328:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914332:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757982.914337:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914341:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787757982.914352:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757982.914356:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787757982.914359:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914364:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.914368:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757982.914374:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787757982.914375:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787757982.914382:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787757982.914383:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914390:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757982.914398:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787757982.914399:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914404:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757982.914414:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659702784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757982.914416:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757982.914423:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757982.914433:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757982.914445:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:0.0:1787757982.914447:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702528:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787757982.914450:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914462:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757982.914463:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.914466:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757982.914472:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.914475:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757982.914481:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914491:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757982.914494:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757982.914500:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757982.914521:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757982.914528:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757982.914532:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757982.914534:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914536:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757982.914540:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914545:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305710094 00000100:00000001:0.0:1787757982.914549:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305710785 00000100:00000001:0.0:1787757982.914553:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305710094 00000100:00000001:0.0:1787757982.914556:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305710785 00000100:00000001:0.0:1787757982.914559:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757982.914561:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914564:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757982.914570:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757982/real 1787757982] req@ffff8ffb45c30700 x1874588659702528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757982.914580:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757982.914582:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757982.914585:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757982.914587:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757982.914590:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914594:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757982.914597:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757982.914604:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659702528/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757982.914619:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914623:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757982.914640:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914643:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757982.914646:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914649:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757982.914671:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757982.914677:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757982.914685:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757982.914690:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757982.914697:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702528:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757982.914705:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757982.914708:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757982.914711:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.914716:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757982.914719:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757982.914723:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757982.914729:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757982.914735:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757982.914738:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757982.914741:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.914745:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757982.914750:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914761:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757982.914765:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787757982.914772:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757982.914798:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757982.914808:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757982.914813:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757982.914816:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914819:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757982.914825:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914831:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305710091 00000100:00000001:0.0:1787757982.914835:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305710785 00000100:00000001:0.0:1787757982.914838:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305710091 00000100:00000001:0.0:1787757982.914841:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305710785 00000100:00000001:0.0:1787757982.914844:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757982.914846:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.914849:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757982.914855:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757982/real 1787757982] req@ffff8ffb45c31500 x1874588659702656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757982.914865:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757982.914867:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757982.914908:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757982.914911:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757982.914914:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914917:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757982.914919:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757982.914925:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659702656/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757982.914934:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914937:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757982.914940:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914942:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757982.914944:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914946:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757982.914959:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757982.914962:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757982.914966:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757982.914969:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757982.914973:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757982.914978:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.914981:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757982.914983:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757982.914985:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757982.914988:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757982.914990:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757982.914992:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757982.914995:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757982.914997:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757982.915000:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757982.915001:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915003:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757982.915005:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757982.915007:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757982.915009:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757982.915010:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757982.915011:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757982.915013:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757982.915015:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757982.915016:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757982.915019:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915021:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915026:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757982.915028:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.915030:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757982.915032:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757982.915034:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757982.915036:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757982.915038:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.915041:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757982.915043:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757982.915046:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757982.915050:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702784:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757982.915053:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757982.915056:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757982.915058:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757982.915060:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757982.915064:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.915071:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757982.915073:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787757982.915077:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757982.915094:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757982.915100:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757982.915102:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757982.915104:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.915106:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757982.915110:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.915114:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305710091 00000100:00000001:0.0:1787757982.915117:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305710785 00000100:00000001:0.0:1787757982.915120:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305710091 00000100:00000001:0.0:1787757982.915123:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305710785 00000100:00000001:0.0:1787757982.915125:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757982.915128:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.915130:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757982.915135:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757982/real 1787757982] req@ffff8ffb45c30a80 x1874588659702784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757982.915145:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757982.915147:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757982.915150:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757982.915152:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757982.915154:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915156:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757982.915157:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757982.915162:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659702784/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757982.915170:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915173:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757982.915176:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915178:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757982.915180:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915182:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757982.915224:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757982.915228:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757982.915232:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757982.915234:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757982.915238:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659702784:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757982.915243:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915246:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757982.915247:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757982.915249:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757982.915252:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757982.915253:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757982.915255:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757982.915257:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757982.915259:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757982.915262:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757982.915264:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915266:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757982.915270:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757982.915272:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757982.915275:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757982.915277:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757983.489340:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757983.489350:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757983.489353:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757983.489359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757983.489364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757983.489369:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757983.489372:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757983.489375:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757983.489378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757983.681407:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757983.681416:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757983.681424:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757983.681431:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757983.681436:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757983.681444:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757983.681448:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757983.681451:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757983.681455:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757983.681459:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757983.681541:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757983.681550:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757983.681557:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757984.513346:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757984.513355:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757984.513358:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757984.513364:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757984.513369:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757984.513373:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757984.513376:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757984.513379:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757984.513383:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757984.705320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757984.705330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757984.705343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757984.705347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757984.705379:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757984.705383:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757984.705386:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757985.537308:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757985.537318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757985.537321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757985.537327:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757985.537332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757985.537336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757985.537339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757985.537342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757985.537346:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757985.729320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757985.729331:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757985.729335:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757985.729340:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757985.729347:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757985.729350:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757985.729353:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757985.729356:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757985.729361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757985.729363:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757985.729501:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757985.729507:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757985.729512:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757986.561300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757986.561309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757986.561312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757986.561318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757986.561323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757986.561329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757986.561332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757986.561336:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757986.561339:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757986.753308:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757986.753316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757986.753324:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757986.753327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757986.753365:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757986.753372:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757986.753377:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757987.585310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757987.585328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757987.585332:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757987.585342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757987.585348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757987.585353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757987.585357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757987.585361:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757987.585366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757987.777332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757987.777340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757987.777348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757987.777350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757987.777449:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757987.777456:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757987.777461:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757988.033325:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757988.033337:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757988.033341:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757988.033345:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757988.033351:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757988.033359:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11042 00000100:00080000:3.0:1787757988.033373:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757988.033377:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757988.033380:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757988.033385:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757988.033388:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757988.033391:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757988.033394:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757988.033397:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757988.033399:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757988.033405:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757988.033408:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757988.033412:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757988.033416:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757988.033420:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033423:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757988.033426:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757988.033439:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757988.033441:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033445:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757988.033449:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757988.033457:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033475:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757988.033492:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659702912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757988.033512:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757988.033520:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033550:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757988.033557:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757988.033564:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757988.033567:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757988.033570:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757988.033577:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757988.033583:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11042 00000100:00080000:3.0:1787757988.033592:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757988.033596:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757988.033598:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757988.033602:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757988.033604:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757988.033607:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787757988.033611:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:3.0:1787757988.033611:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757988.033627:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757988.033630:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757988.033632:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757988.033638:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:3.0:1787757988.033646:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757988.033647:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757988.033657:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033658:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757988.033675:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757988.033681:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757988.033685:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757988.033687:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757988.033697:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757988.033699:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033705:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757988.033713:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757988.033721:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659702912:192.168.201.118@tcp:250:kworker.0 00000008:00000001:3.0:1787757988.033724:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033739:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.033743:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787757988.033752:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757988.033754:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757988.033764:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757988.033764:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757988.033773:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787757988.033778:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757988.033809:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757988.033812:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757988.033817:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033836:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.033845:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757988.033850:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00100000:3.0:1787757988.033856:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659703040/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757988.033859:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757988.033882:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757988.033887:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033894:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757988.033898:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757988.033903:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757988.033906:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757988.033909:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757988.033913:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757988.033918:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11042 00000100:00080000:3.0:1787757988.033925:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757988.033928:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757988.033938:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:0.0:1787757988.033939:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:3.0:1787757988.033953:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757988.033956:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757988.033961:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757988.033968:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757988.033971:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787757988.033973:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757988.033979:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757988.033983:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787757988.033989:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757988.033992:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.033997:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.033999:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787757988.034009:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757988.034017:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757988.034018:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757988.034027:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034028:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305715905 00000100:00000001:3.0:1787757988.034046:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757988.034050:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757988.034055:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034059:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787757988.034059:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757988.034068:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757988.034072:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034075:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305715905 00000100:00000001:3.0:1787757988.034077:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757988.034088:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757988.034092:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034094:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787757988.034103:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787757988.034105:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034109:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034115:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787757988.034119:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659703168/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757988.034133:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757988/real 1787757988] req@ffff8ffb45c33b80 x1874588659702912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757988.034164:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757988.034176:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.034227:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757988.034234:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757988.034239:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757988.034241:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757988.034243:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757988.034247:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757988.034253:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11042 00000100:00080000:3.0:1787757988.034259:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757988.034263:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757988.034269:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757988.034284:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757988.034293:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757988.034301:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787757988.034314:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787757988.034322:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757988.034328:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757988.034349:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787757988.034354:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787757988.034356:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757988.034362:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787757988.034363:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757988.034374:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757988.034381:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787757988.034381:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.034390:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757988.034394:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757988.034397:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757988.034400:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757988.034406:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757988.034408:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034410:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787757988.034410:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757988.034412:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757988.034423:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034425:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757988.034434:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034435:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757988.034446:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787757988.034457:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659703296/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757988.034471:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659702912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787757988.034525:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757988.034532:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757988.034534:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034545:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787757988.034547:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034565:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.034572:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757988.034575:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.034581:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757988.034586:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757988.034594:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757988.034607:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757988.034613:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757988.034622:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659702912:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757988.034633:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.034642:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757988.034646:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757988.034654:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757988.034665:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757988.034671:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757988.034674:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757988.034681:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757988.034684:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757988.034689:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757988.034692:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.034695:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.034708:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757988.034711:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034714:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757988.034717:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757988.034720:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757988.034722:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757988.034724:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757988.034729:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757988.034733:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757988.034737:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757988.034743:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703040:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757988.034749:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757988.034753:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757988.034757:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757988.034759:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757988.034765:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034777:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757988.034808:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:0.0:1787757988.034816:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757988.034852:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757988.034866:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757988.034871:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757988.034875:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034878:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757988.034882:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034889:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305715214 00000100:00000001:0.0:1787757988.034893:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305715905 00000100:00000001:0.0:1787757988.034896:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305715214 00000100:00000001:0.0:1787757988.034900:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305715905 00000100:00000001:0.0:1787757988.034902:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757988.034907:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.034923:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757988.034937:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757988/real 1787757988] req@ffff8ffb45c31c00 x1874588659703040/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757988.034973:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757988.034976:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757988.034981:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757988.034984:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757988.034986:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.034988:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757988.034990:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757988.034998:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659703040/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757988.035008:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035011:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757988.035019:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035022:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757988.035024:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035026:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757988.035049:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757988.035053:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757988.035058:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757988.035061:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757988.035066:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703040:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757988.035071:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757988.035073:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757988.035075:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757988.035080:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757988.035084:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757988.035087:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757988.035092:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757988.035097:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757988.035100:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757988.035106:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757988.035109:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757988.035113:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035122:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757988.035124:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:0.0:1787757988.035130:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757988.035149:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757988.035155:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757988.035158:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757988.035160:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035162:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757988.035166:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035170:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305715211 00000100:00000001:0.0:1787757988.035174:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305715905 00000100:00000001:0.0:1787757988.035177:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305715211 00000100:00000001:0.0:1787757988.035181:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305715905 00000100:00000001:0.0:1787757988.035225:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757988.035228:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035230:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757988.035236:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757988/real 1787757988] req@ffff8ffb45c30e00 x1874588659703168/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757988.035247:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757988.035249:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757988.035254:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757988.035256:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757988.035258:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035260:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757988.035263:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757988.035269:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659703168/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757988.035278:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035281:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757988.035288:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035291:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757988.035294:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035296:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757988.035325:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757988.035329:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757988.035338:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757988.035343:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757988.035349:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757988.035360:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757988.035364:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757988.035369:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757988.035376:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757988.035381:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757988.035386:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757988.035395:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703296:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757988.035403:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757988.035407:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757988.035412:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757988.035417:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757988.035426:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035438:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757988.035443:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787757988.035451:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757988.035495:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757988.035516:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757988.035527:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757988.035534:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035537:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757988.035545:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035552:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305715212 00000100:00000001:0.0:1787757988.035559:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305715905 00000100:00000001:0.0:1787757988.035565:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305715212 00000100:00000001:0.0:1787757988.035569:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305715905 00000100:00000001:0.0:1787757988.035573:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757988.035578:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035582:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757988.035594:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757988/real 1787757988] req@ffff8ffb45c30a80 x1874588659703296/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757988.035606:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757988.035608:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757988.035614:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757988.035621:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757988.035626:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035629:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757988.035631:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757988.035636:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659703296/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757988.035645:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035648:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757988.035650:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035652:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757988.035654:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035656:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757988.035676:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757988.035679:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757988.035682:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757988.035684:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757988.035688:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703296:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757988.035693:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035696:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757988.035698:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757988.035700:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757988.035705:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757988.035707:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757988.035709:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757988.035712:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757988.035714:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757988.035717:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757988.035719:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035721:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757988.035723:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757988.035725:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757988.035727:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757988.035729:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757988.035730:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757988.035733:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757988.035734:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757988.035736:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757988.035738:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035739:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757988.035741:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757988.035742:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757988.035744:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757988.035745:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757988.035746:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757988.035748:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757988.035749:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757988.035751:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757988.035753:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035755:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757988.035759:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757988.035761:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757988.035764:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757988.035766:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757988.609271:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757988.609281:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757988.609284:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757988.609290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757988.609295:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757988.609300:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757988.609303:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757988.609306:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757988.609309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757988.801338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757988.801347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757988.801355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757988.801357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757988.801392:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757988.801395:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757988.801399:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757989.633301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757989.633313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757989.633317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757989.633327:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757989.633333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757989.633338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757989.633342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757989.633345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757989.633349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757989.825341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757989.825349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757989.825356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757989.825358:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757989.825478:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757989.825484:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757989.825489:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757990.657283:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757990.657292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757990.657295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757990.657301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757990.657306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757990.657311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757990.657314:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757990.657318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757990.657321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757990.849316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757990.849325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757990.849334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757990.849337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757990.849444:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757990.849452:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757990.849458:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757991.681288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757991.681298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757991.681301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757991.681307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757991.681312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757991.681316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757991.681320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757991.681323:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757991.681327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757991.873320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757991.873328:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757991.873334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757991.873336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757991.873445:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757991.873451:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757991.873456:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757992.705298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757992.705310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757992.705314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757992.705322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757992.705329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757992.705335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757992.705340:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757992.705345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757992.705350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757992.897327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757992.897339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757992.897352:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757992.897355:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757992.897402:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757992.897407:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757992.897413:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757993.153390:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757993.153407:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787757993.153411:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757993.153418:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757993.153425:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757993.153436:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11047 00000100:00080000:3.0:1787757993.153452:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757993.153458:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757993.153464:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757993.153471:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757993.153476:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757993.153481:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787757993.153486:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787757993.153490:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757993.153493:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757993.153502:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757993.153507:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757993.153511:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757993.153516:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757993.153520:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153523:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787757993.153526:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757993.153540:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757993.153542:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153546:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757993.153550:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757993.153555:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153569:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757993.153581:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659703424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787757993.153595:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757993.153601:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153623:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757993.153628:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757993.153633:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787757993.153635:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757993.153638:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757993.153641:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757993.153644:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11047 00000100:00080000:3.0:1787757993.153649:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787757993.153652:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757993.153654:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787757993.153657:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757993.153659:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757993.153662:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787757993.153666:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787757993.153669:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757993.153671:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757993.153677:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757993.153681:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757993.153683:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757993.153717:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757993.153722:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153724:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757993.153732:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787757993.153732:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757993.153741:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.153742:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787757993.153745:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.153749:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757993.153755:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757993.153756:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757993.153759:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757993.153762:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153768:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757993.153770:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787757993.153771:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757993.153778:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757993.153780:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153793:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.153799:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757993.153802:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659703552/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757993.153814:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659703424:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787757993.153829:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787757993.153830:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787757993.153838:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.153840:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757993.153845:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153848:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.153850:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787757993.153857:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787757993.153863:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757993.153864:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787757993.153872:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757993.153875:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757993.153880:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787757993.153885:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11047 00000100:00000001:0.0:1787757993.153890:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787757993.153892:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757993.153897:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787757993.153899:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757993.153905:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757993.153906:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787757993.153914:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757993.153917:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787757993.153920:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787757993.153924:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787757993.153927:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757993.153929:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757993.153934:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757993.153936:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757993.153938:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757993.153943:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757993.153946:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153949:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757993.153952:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787757993.153955:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153958:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757993.153965:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787757993.153967:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153969:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757993.153970:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787757993.153972:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.153977:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.153985:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757993.153986:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659703680/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757993.154002:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305721025 00000100:00000001:3.0:1787757993.154004:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757993.154013:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.154019:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787757993.154023:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757993.154029:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787757993.154029:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787757993.154035:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787757993.154041:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787757993.154045:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787757993.154048:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305721025 00000100:00080000:3.0:1787757993.154050:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11047 00000100:00080000:3.0:1787757993.154060:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757993.154063:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787757993.154064:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757993.154071:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757993.154075:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787757993.154077:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787757993.154083:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787757993.154086:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757993.154087:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757993.154095:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757993.154097:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787757993.154107:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787757993.154108:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787757993.154114:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787757993.154122:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787757993.154125:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757993.154127:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787757993.154131:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757993.154133:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154137:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787757993.154137:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787757993.154141:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757993.154143:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000008:00000001:3.0:1787757993.154150:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154153:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787757993.154158:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154160:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757993.154166:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757993.154169:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787757993.154170:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757993.154173:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787757993.154176:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.154181:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787757993.154222:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787757993.154226:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787757993.154232:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787757993.154234:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.154244:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.154251:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757993.154255:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703552:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757993.154265:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757993.154269:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757993.154275:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787757993.154277:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659703808/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757993.154279:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757993.154287:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154302:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757993.154307:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787757993.154321:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787757993.154333:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787757993.154339:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787757993.154346:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154369:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154382:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305720334 00000100:00000001:0.0:1787757993.154389:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305721025 00000100:00000001:0.0:1787757993.154396:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305720334 00000100:00000001:0.0:1787757993.154401:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305721025 00000100:00000001:0.0:1787757993.154405:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757993.154409:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154415:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757993.154418:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757993.154420:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.154425:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757993.154429:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757993.154432:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757993.154441:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703680:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757993.154446:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757993.154448:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757993.154450:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.154453:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757993.154457:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154466:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757993.154467:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787757993.154473:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787757993.154489:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154493:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305720331 00000100:00000001:0.0:1787757993.154497:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305721025 00000100:00000001:0.0:1787757993.154499:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305720331 00000100:00000001:0.0:1787757993.154502:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305721025 00000100:00000001:0.0:1787757993.154505:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757993.154507:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154510:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154513:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757993.154517:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757993.154519:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787757993.154522:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757993.154523:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757993.154525:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757993.154527:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757993.154529:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.154531:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757993.154533:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757993.154536:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757993.154540:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703808:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757993.154544:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757993.154546:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757993.154548:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757993.154550:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757993.154553:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154557:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757993.154559:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787757993.154563:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787757993.154573:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154576:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305720331 00000100:00000001:0.0:1787757993.154579:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305721025 00000100:00000001:0.0:1787757993.154582:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305720331 00000100:00000001:0.0:1787757993.154585:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305721025 00000100:00000001:0.0:1787757993.154587:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757993.154589:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154592:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154594:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757993.154597:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757993.154599:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787757993.154601:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757993.154602:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757993.154604:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757993.154607:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757993.729296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757993.729306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757993.729309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757993.729316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757993.729320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757993.729325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757993.729329:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757993.729332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757993.729336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757993.921388:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757993.921399:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757993.921413:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757993.921421:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757993.921428:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757993.921436:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757993.921444:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757993.921450:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757993.921455:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757993.921463:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757993.921588:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757993.921595:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757993.921600:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757994.753273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757994.753282:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757994.753285:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757994.753292:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757994.753296:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757994.753301:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757994.753305:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757994.753309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757994.753313:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757994.945329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757994.945339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757994.945349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757994.945352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757994.945484:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787757994.945490:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787757994.945495:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787757995.777295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757995.777305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757995.777307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757995.777314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757995.777318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757995.777323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757995.777327:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757995.777330:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757995.777334:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757995.969318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757995.969328:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787757995.969332:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787757995.969336:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757995.969344:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787757995.969347:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757995.969349:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757995.969353:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757995.969358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757995.969359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757995.969446:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757995.969452:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757995.969457:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757996.225475:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1787757996.225542:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757996.225568:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757996.225591:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757996.225623:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757996.225667:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757996.225672:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787757996.225691:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757996.225703:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757996.225712:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757996.225717:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757996.225718:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787757996.225725:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757996.225727:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757996.225736:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000800:00000100:1.0:1787757996.225745:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757996.225756:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:0.0:1787757996.225759:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757993/real 1787757996] req@ffff8ffb45c30700 x1874588659703552/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787757996.225764:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787757996.225769:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757996.225777:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787757996.225792:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787757996.225804:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787757996.225809:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000400:00000001:0.0:1787757996.225812:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787757996.225813:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757996.225820:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000400:00000001:0.0:1787757996.225822:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757996.225839:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757996.225845:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757996.225848:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.225853:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757996.225857:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757996.225869:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659703552/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757996.225885:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.225891:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757996.225897:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.225901:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757996.225903:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.225908:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757996.225932:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757996.225939:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757996.225949:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757996.225954:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757996.225961:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703552:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757996.225970:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757996.225978:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757993/real 1787757996] req@ffff8ffb45c32a00 x1874588659703680/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757996.225993:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757996.225996:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757996.226002:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757996.226006:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757996.226009:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226012:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757996.226015:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757996.226023:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659703680/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757996.226038:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226042:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757996.226047:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226050:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757996.226053:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226056:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757996.226077:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757996.226082:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757996.226090:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757996.226094:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757996.226101:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703680:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757996.226110:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757996.226119:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757993/real 1787757996] req@ffff8ffb45c31500 x1874588659703424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757996.226134:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757996.226137:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757996.226143:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757996.226146:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757996.226148:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226152:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757996.226155:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757996.226162:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659703424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757996.226175:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226179:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757996.226234:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226238:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757996.226240:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226243:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757996.226246:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757996.226251:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757996.226257:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757996.226261:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757996.226268:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659703424:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757996.226275:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757996.226279:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757996.226283:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757996.226288:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757996.226297:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757996.226301:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757996.226304:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757996.226310:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757996.226314:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757996.226319:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757996.226321:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226325:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757996.226327:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757996.226329:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757996.226332:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757996.226334:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757996.226336:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757996.226340:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757996.226342:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757996.226346:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757996.226348:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226362:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757996.226366:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757996.226368:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757996.226372:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757996.226374:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757996.226377:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757996.226380:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757996.226382:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757996.226385:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757996.226387:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226392:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757996.226396:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757996.226398:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757996.226401:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757996.226412:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757993/real 1787757996] req@ffff8ffb45c30000 x1874588659703808/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757996.226431:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757996.226434:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757996.226439:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757996.226442:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757996.226445:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226449:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757996.226451:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757996.226459:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659703808/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757996.226472:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226476:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757996.226481:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226485:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757996.226487:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226491:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757996.226513:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757996.226518:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757996.226526:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757996.226531:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757996.226537:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659703808:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757996.226546:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226550:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757996.226552:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757996.226555:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757996.226560:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757996.226562:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757996.226565:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757996.226569:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757996.226572:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757996.226575:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757996.226580:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226585:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757996.226597:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757996.226601:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757996.226605:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757996.226607:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787757996.801353:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787757996.801365:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787757996.801368:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787757996.801375:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787757996.801380:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787757996.801385:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757996.801389:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787757996.801392:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787757996.801396:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757996.993296:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757996.993304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757996.993312:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757996.993315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787757996.993348:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787757996.993351:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787757996.993355:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757997.825312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757997.825321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757997.825324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757997.825330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757997.825335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757997.825340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757997.825344:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757997.825347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757997.825350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757998.017330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757998.017339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757998.017346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757998.017348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787757998.017451:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787757998.017460:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787757998.017469:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757998.273397:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757998.273414:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787757998.273418:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757998.273424:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757998.273431:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757998.273442:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11053 00000100:00080000:2.0:1787757998.273460:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757998.273465:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757998.273469:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757998.273475:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757998.273479:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757998.273483:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787757998.273487:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787757998.273489:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757998.273491:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757998.273498:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757998.273501:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757998.273504:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757998.273510:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757998.273514:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.273518:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787757998.273521:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757998.273539:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757998.273542:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.273547:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757998.273551:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757998.273558:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.273577:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757998.273593:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659703936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787757998.273644:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757998.273653:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.273690:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757998.273698:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757998.273705:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787757998.273709:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757998.273713:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757998.273719:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757998.273725:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11053 00000100:00080000:2.0:1787757998.273734:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757998.273739:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757998.273742:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787757998.273749:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757998.273752:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757998.273756:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787757998.273762:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787757998.273766:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757998.273769:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787757998.273778:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787757998.273783:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.273788:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787757998.273788:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787757998.273806:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787757998.273809:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757998.273819:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.273823:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757998.273827:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:2.0:1787757998.273832:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757998.273837:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:2.0:1787757998.273844:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.273847:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.273854:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757998.273864:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787757998.273870:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787757998.273875:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787757998.273878:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.273895:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787757998.273896:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757998.273900:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757998.273910:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757998.273920:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659703936:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787757998.273930:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.273940:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757998.273952:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:2.0:1787757998.273954:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659704064/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787757998.273961:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757998.273976:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787757998.273990:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787757998.273992:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.274001:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.274026:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757998.274032:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:0.0:1787757998.274048:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787757998.274052:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787757998.274062:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757998.274070:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787757998.274075:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757998.274080:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787757998.274088:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787757998.274097:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11053 00000100:00080000:2.0:1787757998.274107:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787757998.274113:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757998.274117:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:0.0:1787757998.274132:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787757998.274133:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787757998.274142:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787757998.274150:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787757998.274157:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787757998.274162:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787757998.274164:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787757998.274172:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757998.274177:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:2.0:1787757998.274281:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757998.274291:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.274293:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.274307:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787757998.274308:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787757998.274323:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757998.274327:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757998.274331:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.274341:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787757998.274342:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305726146 00000008:00100000:2.0:1787757998.274349:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787757998.274360:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.274364:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.274366:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787757998.274380:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787757998.274384:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305726146 02000000:00000001:2.0:1787757998.274389:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.274399:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787757998.274402:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787757998.274404:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757998.274411:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.274429:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.274430:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757998.274441:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787757998.274446:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659704192/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757998.274452:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757998.274479:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757998/real 1787757998] req@ffff8ffb47a38380 x1874588659703936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787757998.274500:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:0.0:1787757998.274507:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787757998.274511:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787757998.274517:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787757998.274525:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.274531:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:2.0:1787757998.274543:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787757998.274552:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:2.0:1787757998.274557:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787757998.274563:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757998.274566:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787757998.274573:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787757998.274576:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757998.274590:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787757998.274591:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:0.0:1787757998.274636:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659703936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757998.274655:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.274661:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757998.274667:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.274670:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757998.274673:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.274678:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757998.274683:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757998.274690:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757998.274698:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757998.274704:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757998.274713:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659703936:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787757998.274723:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.274728:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757998.274731:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757998.274736:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757998.274745:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757998.274749:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757998.274753:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757998.274760:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757998.274764:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757998.274770:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757998.274773:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.274778:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.274790:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:2.0:1787757998.274794:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11053 00000100:00000001:0.0:1787757998.274798:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.274813:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787757998.274814:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787757998.274820:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787757998.274825:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787757998.274830:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787757998.274834:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787757998.274840:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000020:00000001:2.0:1787757998.274845:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787757998.274849:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787757998.274854:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787757998.274863:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:2.0:1787757998.274864:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:0.0:1787757998.274874:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787757998.274877:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787757998.274895:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787757998.274898:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.274901:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787757998.274915:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704064:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1787757998.274932:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787757998.274935:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787757998.274946:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757998.274947:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757998.274955:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757998.274958:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787757998.274965:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787757998.274972:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757998.274980:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787757998.274983:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.274993:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787757998.274998:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787757998.275005:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00000001:2.0:1787757998.275005:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275017:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:2.0:1787757998.275018:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275039:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787757998.275041:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787757998.275048:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.275054:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787757998.275058:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787757998.275063:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.275076:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787757998.275091:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1787757998.275094:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659704320/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787757998.275111:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757998.275120:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757998.275127:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275140:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757998.275150:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787757998.275154:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787757998.275164:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275165:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305725454 00000100:00000001:2.0:1787757998.275179:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275181:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305726146 00000100:00000001:0.0:1787757998.275251:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305725455 00000100:00000001:0.0:1787757998.275256:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305726146 00000100:00000001:0.0:1787757998.275260:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757998.275263:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275267:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757998.275277:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757998/real 1787757998] req@ffff8ffb47a38000 x1874588659704064/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757998.275298:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757998.275301:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757998.275309:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757998.275313:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757998.275316:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275320:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757998.275323:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757998.275333:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659704064/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757998.275350:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275355:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757998.275362:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275367:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757998.275371:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275376:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757998.275401:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757998.275406:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757998.275412:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757998.275416:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757998.275420:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704064:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787757998.275426:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757998.275429:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757998.275431:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757998.275434:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757998.275437:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757998.275440:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757998.275444:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757998.275452:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757998.275455:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757998.275458:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757998.275460:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757998.275466:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275475:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757998.275477:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787757998.275482:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757998.275503:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757998.275511:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757998.275514:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757998.275516:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275518:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757998.275522:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275527:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305725452 00000100:00000001:0.0:1787757998.275531:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305726146 00000100:00000001:0.0:1787757998.275535:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305725452 00000100:00000001:0.0:1787757998.275540:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305726146 00000100:00000001:0.0:1787757998.275545:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757998.275548:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275552:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757998.275560:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757998/real 1787757998] req@ffff8ffb47a3b480 x1874588659704192/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757998.275577:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757998.275580:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757998.275586:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757998.275589:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757998.275593:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275597:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757998.275599:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757998.275630:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659704192/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757998.275659:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275664:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757998.275669:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275673:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757998.275675:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275677:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757998.275690:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757998.275694:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757998.275698:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757998.275701:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757998.275706:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757998.275711:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275714:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757998.275716:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757998.275718:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757998.275723:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757998.275725:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757998.275728:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757998.275734:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757998.275737:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757998.275741:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757998.275744:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275747:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757998.275750:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757998.275752:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757998.275755:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757998.275758:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757998.275760:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757998.275763:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757998.275765:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757998.275769:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757998.275771:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275775:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.275784:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757998.275788:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275793:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757998.275796:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757998.275799:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787757998.275803:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757998.275805:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757998.275811:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787757998.275814:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787757998.275818:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757998.275826:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704320:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757998.275834:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787757998.275837:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787757998.275839:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787757998.275842:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787757998.275846:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275855:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787757998.275857:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787757998.275862:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787757998.275886:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787757998.275895:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787757998.275900:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757998.275903:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275906:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787757998.275911:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275918:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305725452 00000100:00000001:0.0:1787757998.275923:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305726146 00000100:00000001:0.0:1787757998.275927:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305725452 00000100:00000001:0.0:1787757998.275932:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305726146 00000100:00000001:0.0:1787757998.275936:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787757998.275940:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.275944:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787757998.275953:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757998/real 1787757998] req@ffff8ffb47a39180 x1874588659704320/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787757998.275984:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787757998.275986:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787757998.275992:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787757998.275995:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787757998.275998:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.276002:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757998.276005:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787757998.276013:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659704320/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787757998.276027:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.276032:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787757998.276037:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.276040:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787757998.276043:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.276046:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787757998.276068:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787757998.276073:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757998.276080:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787757998.276086:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787757998.276091:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704320:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787757998.276096:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.276099:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787757998.276101:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787757998.276103:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787757998.276107:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787757998.276109:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787757998.276111:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787757998.276114:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787757998.276115:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787757998.276120:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787757998.276121:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.276124:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.276129:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757998.276131:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.276134:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787757998.276136:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787757998.849293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757998.849299:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757998.849301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757998.849305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757998.849308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757998.849310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.849312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757998.849315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757998.849317:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757999.041320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757999.041326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787757999.041333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787757999.041334:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787757999.041427:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787757999.041436:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787757999.041442:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:0.0:1787757999.873315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1787757999.873323:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787757999.873325:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787757999.873331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:0.0:1787757999.873336:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787757999.873341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757999.873345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787757999.873348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787757999.873351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758000.065259:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758000.065264:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758000.065270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758000.065271:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758000.065355:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758000.065358:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758000.065361:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758000.897269:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758000.897276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758000.897278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758000.897282:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758000.897286:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758000.897289:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758000.897291:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758000.897293:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758000.897295:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758001.089263:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758001.089269:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758001.089273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758001.089274:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758001.089290:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758001.089292:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758001.089294:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758001.921260:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758001.921268:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758001.921271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758001.921277:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758001.921281:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758001.921285:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758001.921289:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758001.921291:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758001.921294:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758002.113270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758002.113276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758002.113280:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758002.113281:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758002.113329:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758002.113332:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758002.113335:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758002.945294:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758002.945300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758002.945302:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758002.945306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758002.945309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758002.945312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758002.945315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758002.945317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758002.945319:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758003.137272:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758003.137278:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758003.137282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758003.137285:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758003.137287:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758003.137291:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758003.137294:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758003.137296:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758003.137297:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758003.137298:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758003.137334:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758003.137338:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758003.137341:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758003.393249:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758003.393257:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758003.393259:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758003.393262:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758003.393265:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758003.393271:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11058 00000100:00080000:2.0:1787758003.393280:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758003.393282:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758003.393284:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758003.393287:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758003.393289:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758003.393292:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758003.393293:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758003.393295:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758003.393296:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758003.393299:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758003.393301:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758003.393303:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758003.393305:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758003.393307:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393309:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758003.393311:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758003.393320:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758003.393321:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393324:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758003.393326:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758003.393330:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393338:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758003.393346:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659704448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758003.393354:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758003.393357:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393371:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758003.393373:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758003.393376:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758003.393377:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758003.393378:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758003.393380:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758003.393381:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11058 00000100:00080000:2.0:1787758003.393383:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758003.393385:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758003.393385:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758003.393387:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758003.393388:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758003.393390:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758003.393391:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758003.393393:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758003.393394:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758003.393396:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758003.393398:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758003.393399:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758003.393403:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758003.393405:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393406:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758003.393410:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758003.393413:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393414:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393415:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787758003.393417:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758003.393420:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787758003.393420:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393421:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758003.393424:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787758003.393424:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758003.393428:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393429:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758003.393432:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393433:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758003.393438:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659704576/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758003.393443:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787758003.393444:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758003.393447:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758003.393448:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393453:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758003.393454:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758003.393457:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758003.393461:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659704448:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787758003.393462:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758003.393465:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758003.393467:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758003.393469:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787758003.393470:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787758003.393473:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:2.0:1787758003.393474:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11058 02000000:00000001:0.0:1787758003.393476:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758003.393478:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758003.393481:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758003.393481:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758003.393484:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758003.393486:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758003.393486:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787758003.393488:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758003.393488:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758003.393493:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758003.393495:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758003.393496:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758003.393498:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758003.393501:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758003.393502:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758003.393504:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:2.0:1787758003.393504:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758003.393506:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393507:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:2.0:1787758003.393508:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758003.393523:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758003.393525:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00000001:2.0:1787758003.393527:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393529:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758003.393534:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758003.393535:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393536:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758003.393537:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758003.393538:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393543:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758003.393546:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659704704/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758003.393551:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758003.393553:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393555:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758003.393557:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758003.393560:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787758003.393563:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758003.393563:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758003.393564:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758003.393568:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758003.393571:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11058 00000100:00080000:2.0:1787758003.393573:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758003.393574:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758003.393575:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758003.393577:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:2.0:1787758003.393577:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758003.393581:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758003.393582:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758003.393583:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:2.0:1787758003.393584:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758003.393587:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758003.393589:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393589:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758003.393595:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:2.0:1787758003.393595:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758003.393600:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758003.393602:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758003.393604:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758003.393605:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758003.393606:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393611:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758003.393612:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758003.393613:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305731265 00000008:00000001:2.0:1787758003.393614:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393618:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758003.393622:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758003.393623:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393625:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758003.393626:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758003.393627:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787758003.393627:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393630:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758003.393633:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659704832/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758003.393640:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758003.393643:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758003.393645:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393648:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305731265 00000100:00000001:0.0:1787758003.393655:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758003.393667:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758003.393670:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393672:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758003.393679:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758003/real 1787758003] req@ffff8ffb47a3a300 x1874588659704448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758003.393691:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758003.393693:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758003.393697:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758003.393700:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758003.393701:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393704:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758003.393706:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758003.393710:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659704448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758003.393718:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393721:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758003.393724:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393725:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758003.393727:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393729:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758003.393731:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758003.393734:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758003.393739:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758003.393742:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758003.393746:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659704448:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758003.393751:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393753:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758003.393755:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758003.393758:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758003.393761:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758003.393763:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758003.393765:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758003.393769:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758003.393771:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758003.393775:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758003.393776:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393779:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393785:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758003.393787:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393789:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758003.393791:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758003.393793:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758003.393794:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758003.393795:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758003.393797:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758003.393808:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758003.393811:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758003.393814:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704576:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758003.393818:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758003.393820:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758003.393822:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758003.393823:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758003.393825:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393830:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758003.393831:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:0.0:1787758003.393834:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758003.393844:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758003.393848:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758003.393849:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758003.393851:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393852:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758003.393854:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393856:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305730573 00000100:00000001:0.0:1787758003.393858:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305731265 00000100:00000001:0.0:1787758003.393860:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305730573 00000100:00000001:0.0:1787758003.393862:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305731265 00000100:00000001:0.0:1787758003.393863:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758003.393865:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393866:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758003.393869:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758003/real 1787758003] req@ffff8ffb47a38700 x1874588659704576/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758003.393874:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758003.393875:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758003.393876:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758003.393877:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758003.393879:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393880:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758003.393881:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758003.393883:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659704576/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758003.393887:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393889:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758003.393891:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393892:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758003.393893:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393894:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758003.393903:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758003.393905:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758003.393907:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758003.393908:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758003.393910:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704576:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758003.393913:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758003.393914:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758003.393915:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758003.393917:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758003.393918:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758003.393919:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758003.393921:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758003.393923:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758003.393924:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758003.393925:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758003.393926:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758003.393927:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393930:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758003.393931:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787758003.393934:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758003.393940:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758003.393942:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758003.393943:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758003.393944:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393945:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758003.393947:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393949:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305730570 00000100:00000001:0.0:1787758003.393951:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305731265 00000100:00000001:0.0:1787758003.393952:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305730570 00000100:00000001:0.0:1787758003.393954:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305731265 00000100:00000001:0.0:1787758003.393955:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758003.393956:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.393957:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758003.393959:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758003/real 1787758003] req@ffff8ffb47a39180 x1874588659704704/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758003.393964:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758003.393965:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758003.393966:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758003.393967:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758003.393967:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393969:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758003.393970:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758003.393972:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659704704/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758003.393977:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393978:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758003.393979:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393980:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758003.393981:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.393982:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758003.393990:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758003.393992:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758003.393993:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758003.393995:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758003.393996:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758003.393998:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758003.393999:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758003.394000:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758003.394001:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758003.394003:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758003.394004:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758003.394006:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704832:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758003.394008:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758003.394008:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758003.394009:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758003.394010:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758003.394012:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.394014:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758003.394015:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787758003.394017:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758003.394022:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758003.394025:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758003.394026:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758003.394027:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.394028:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758003.394030:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.394031:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305730570 00000100:00000001:0.0:1787758003.394033:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305731265 00000100:00000001:0.0:1787758003.394034:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305730570 00000100:00000001:0.0:1787758003.394036:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305731265 00000100:00000001:0.0:1787758003.394038:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758003.394039:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.394040:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758003.394042:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758003/real 1787758003] req@ffff8ffb47a3b480 x1874588659704832/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758003.394046:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758003.394047:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758003.394048:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758003.394049:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758003.394050:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394051:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758003.394052:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758003.394054:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659704832/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758003.394058:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394059:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758003.394061:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394062:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758003.394063:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394064:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758003.394072:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758003.394074:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758003.394076:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758003.394077:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758003.394079:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659704832:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758003.394082:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394083:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758003.394083:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758003.394084:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758003.394086:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758003.394087:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758003.394087:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758003.394089:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758003.394090:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758003.394092:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758003.394093:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394094:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758003.394094:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758003.394095:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758003.394097:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758003.394098:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758003.394099:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758003.394099:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758003.394100:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758003.394102:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758003.394103:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394104:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758003.394104:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758003.394105:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758003.394106:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758003.394107:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758003.394107:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758003.394108:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758003.394109:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758003.394110:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758003.394111:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394112:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758003.394114:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758003.394115:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758003.394116:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758003.394117:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758003.969278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758003.969284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758003.969285:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758003.969289:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758003.969292:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758003.969295:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758003.969297:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758003.969300:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758003.969302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758004.161258:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758004.161264:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758004.161269:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758004.161271:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758004.161317:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758004.161321:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758004.161324:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758004.993259:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758004.993265:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758004.993267:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758004.993271:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758004.993273:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758004.993277:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758004.993279:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758004.993281:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758004.993283:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758005.185259:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758005.185266:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758005.185268:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758005.185271:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758005.185275:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758005.185276:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758005.185278:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758005.185279:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758005.185282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758005.185283:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758005.185299:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758005.185301:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758005.185303:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758006.017254:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758006.017260:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758006.017261:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758006.017265:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758006.017268:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758006.017271:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758006.017273:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758006.017275:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758006.017277:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758006.209261:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758006.209267:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758006.209272:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758006.209274:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758006.209318:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758006.209321:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758006.209324:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758007.041266:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758007.041272:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758007.041274:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758007.041279:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758007.041283:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758007.041286:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758007.041289:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758007.041292:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758007.041295:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758007.233271:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758007.233279:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758007.233287:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758007.233289:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758007.233360:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758007.233364:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758007.233383:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758008.065278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758008.065288:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758008.065291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758008.065296:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758008.065300:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758008.065304:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758008.065307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758008.065310:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758008.065312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758008.257297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758008.257306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758008.257315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758008.257317:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758008.257499:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758008.257507:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758008.257514:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758008.513364:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758008.513377:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758008.513380:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758008.513384:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758008.513390:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758008.513399:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11063 00000100:00080000:3.0:1787758008.513414:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758008.513418:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758008.513448:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758008.513456:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758008.513459:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758008.513475:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758008.513478:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758008.513481:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758008.513483:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758008.513489:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758008.513492:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758008.513495:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758008.513500:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758008.513504:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513507:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758008.513510:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758008.513523:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758008.513525:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513529:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758008.513532:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758008.513537:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513551:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758008.513563:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659704960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758008.513577:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758008.513582:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513604:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758008.513609:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758008.513613:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758008.513615:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758008.513618:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758008.513622:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758008.513626:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11063 00000100:00080000:3.0:1787758008.513631:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758008.513634:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758008.513637:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758008.513640:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758008.513642:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758008.513645:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758008.513648:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758008.513650:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758008.513652:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758008.513656:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758008.513659:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758008.513661:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758008.513669:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758008.513672:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513675:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758008.513682:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787758008.513683:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758008.513691:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.513692:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758008.513695:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.513699:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758008.513703:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758008.513706:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758008.513708:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.513712:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513714:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758008.513720:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758008.513725:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758008.513726:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.513732:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758008.513735:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.513748:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758008.513757:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659705088/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758008.513760:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659704960:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758008.513779:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758008.513786:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758008.513788:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758008.513790:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.513794:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513798:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.513800:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787758008.513805:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758008.513811:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758008.513812:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758008.513820:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758008.513822:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758008.513827:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758008.513832:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11063 00000100:00000001:0.0:1787758008.513836:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787758008.513838:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758008.513844:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:3.0:1787758008.513845:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758008.513852:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758008.513863:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787758008.513864:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758008.513869:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758008.513873:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758008.513876:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758008.513879:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758008.513880:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758008.513885:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758008.513889:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758008.513890:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758008.513894:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758008.513898:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513901:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758008.513904:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758008.513907:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513909:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758008.513916:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000800:00000100:0.0:1787758008.513917:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787758008.513920:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513925:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758008.513928:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758008.513931:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.513936:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758008.513939:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.513947:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00100000:3.0:1787758008.513951:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659705216/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758008.513953:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.513962:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758008.513971:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.513972:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758008.513979:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.513980:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305736385 00000100:00000001:3.0:1787758008.513986:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758008.513991:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758008.513995:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758008.513997:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758008.513998:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758008.514004:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758008.514009:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758008.514011:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305736385 00000100:00080000:3.0:1787758008.514014:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11063 00000100:00000001:0.0:1787758008.514024:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787758008.514024:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758008.514027:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758008.514033:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758008.514034:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758008.514039:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758008.514042:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758008.514044:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758008.514051:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:3.0:1787758008.514052:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758008.514059:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758008.514062:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:0.0:1787758008.514068:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758008/real 1787758008] req@ffff8ffb45c33800 x1874588659704960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758008.514068:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758008.514076:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758008.514087:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758008.514098:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000400:00000001:0.0:1787758008.514102:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758008.514107:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787758008.514108:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758008.514112:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787758008.514119:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514120:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758008.514125:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758008.514129:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758008.514132:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00100000:3.0:1787758008.514137:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758008.514140:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787758008.514151:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.514167:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758008.514171:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659704960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758008.514180:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758008.514225:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.514229:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758008.514232:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758008.514236:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758008.514245:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758008.514262:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659705344/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758008.514264:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514277:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758008.514284:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758008.514286:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758008.514291:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514293:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758008.514299:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514300:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514307:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758008.514313:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758008.514318:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758008.514325:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758008.514329:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758008.514336:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659704960:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758008.514343:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514347:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758008.514349:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758008.514352:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758008.514357:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758008.514360:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758008.514363:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758008.514367:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758008.514369:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758008.514373:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758008.514375:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514378:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514386:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758008.514389:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514391:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758008.514394:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758008.514396:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758008.514398:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758008.514400:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.514403:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758008.514406:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758008.514409:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758008.514413:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705088:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758008.514417:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758008.514446:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758008.514450:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.514452:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758008.514466:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514475:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758008.514477:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787758008.514483:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758008.514502:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758008.514509:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758008.514512:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758008.514515:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514517:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758008.514521:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514526:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305735694 00000100:00000001:0.0:1787758008.514529:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305736385 00000100:00000001:0.0:1787758008.514533:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305735694 00000100:00000001:0.0:1787758008.514536:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305736385 00000100:00000001:0.0:1787758008.514538:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758008.514541:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514543:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758008.514549:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758008/real 1787758008] req@ffff8ffb45c31180 x1874588659705088/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758008.514559:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758008.514561:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758008.514563:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758008.514570:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758008.514572:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514574:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758008.514576:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758008.514581:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659705088/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758008.514589:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514592:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758008.514596:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514598:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758008.514600:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514602:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758008.514616:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758008.514619:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758008.514623:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758008.514626:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758008.514630:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705088:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758008.514635:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758008.514636:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758008.514638:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.514641:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758008.514643:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758008.514645:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758008.514649:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758008.514653:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758008.514656:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758008.514658:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.514660:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758008.514663:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514681:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758008.514682:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:0.0:1787758008.514687:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758008.514699:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758008.514703:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758008.514706:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758008.514708:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514710:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758008.514713:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514717:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305735691 00000100:00000001:0.0:1787758008.514720:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305736385 00000100:00000001:0.0:1787758008.514723:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305735691 00000100:00000001:0.0:1787758008.514726:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305736385 00000100:00000001:0.0:1787758008.514729:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758008.514731:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514734:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758008.514738:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758008/real 1787758008] req@ffff8ffb45c33480 x1874588659705216/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758008.514747:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758008.514749:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758008.514751:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758008.514753:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758008.514754:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514756:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758008.514758:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758008.514762:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659705216/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758008.514771:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514773:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758008.514776:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514778:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758008.514780:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514782:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758008.514793:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758008.514796:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758008.514799:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758008.514802:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758008.514806:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758008.514811:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758008.514812:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758008.514815:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.514817:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758008.514819:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758008.514822:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758008.514825:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705344:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758008.514829:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758008.514830:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758008.514832:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758008.514834:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758008.514837:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514841:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758008.514843:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787758008.514847:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758008.514858:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758008.514862:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758008.514864:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758008.514866:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514868:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758008.514871:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514874:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305735691 00000100:00000001:0.0:1787758008.514877:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305736385 00000100:00000001:0.0:1787758008.514880:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305735691 00000100:00000001:0.0:1787758008.514883:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305736385 00000100:00000001:0.0:1787758008.514885:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758008.514888:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.514890:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758008.514894:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758008/real 1787758008] req@ffff8ffb45c30000 x1874588659705344/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758008.514903:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758008.514904:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758008.514906:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758008.514908:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758008.514910:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514920:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758008.514923:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758008.514927:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659705344/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758008.514936:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514938:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758008.514941:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514943:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758008.514945:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514947:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758008.514959:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758008.514962:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758008.514965:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758008.514967:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758008.514971:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705344:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758008.514975:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514977:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758008.514979:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758008.514981:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758008.514984:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758008.514986:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758008.514987:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758008.514990:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758008.514992:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758008.514994:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758008.514996:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.514998:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758008.515000:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758008.515001:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758008.515003:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758008.515005:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758008.515006:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758008.515008:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758008.515009:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758008.515011:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758008.515013:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.515015:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758008.515017:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758008.515018:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758008.515021:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758008.515022:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758008.515024:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758008.515025:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758008.515027:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758008.515029:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758008.515030:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.515032:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758008.515036:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758008.515038:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758008.515040:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758008.515042:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758009.089320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758009.089348:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758009.089352:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758009.089360:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758009.089365:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758009.089371:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758009.089374:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758009.089379:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758009.089383:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758009.281344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758009.281353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758009.281366:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758009.281369:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758009.281441:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758009.281445:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758009.281450:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758010.113281:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758010.113294:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758010.113298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758010.113306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758010.113311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758010.113317:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758010.113321:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758010.113326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758010.113330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758010.305309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758010.305321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758010.305334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758010.305337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758010.305410:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758010.305417:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758010.305423:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758011.137249:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758011.137257:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758011.137260:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758011.137266:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758011.137270:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758011.137275:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758011.137277:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758011.137280:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758011.137282:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758011.329321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758011.329332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758011.329342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758011.329345:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758011.329482:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758011.329491:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758011.329497:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758012.161318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758012.161332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758012.161337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758012.161348:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758012.161355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758012.161423:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758012.161427:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758012.161432:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758012.161436:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758012.353264:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758012.353272:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758012.353278:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758012.353280:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758012.353313:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758012.353317:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758012.353320:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758013.185244:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758013.185249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758013.185251:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758013.185255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758013.185258:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758013.185262:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758013.185263:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758013.185265:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758013.185268:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758013.377270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758013.377277:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758013.377281:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758013.377284:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758013.377287:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758013.377290:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758013.377292:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758013.377294:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758013.377296:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758013.377297:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758013.377314:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758013.377316:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758013.377318:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758013.633320:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758013.633328:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758013.633357:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758013.633361:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758013.633365:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758013.633370:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11068 00000100:00080000:2.0:1787758013.633380:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758013.633382:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758013.633384:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758013.633387:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758013.633389:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758013.633392:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758013.633394:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758013.633396:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758013.633397:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758013.633400:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758013.633401:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758013.633403:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758013.633406:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758013.633408:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633410:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758013.633412:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758013.633422:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758013.633424:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633426:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758013.633429:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758013.633432:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633442:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758013.633449:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659705472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758013.633457:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758013.633460:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633473:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758013.633476:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758013.633478:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758013.633480:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758013.633481:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758013.633483:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758013.633484:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11068 00000100:00080000:2.0:1787758013.633486:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758013.633488:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758013.633489:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758013.633491:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758013.633492:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758013.633493:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758013.633495:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758013.633497:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758013.633497:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758013.633500:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758013.633502:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758013.633503:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758013.633507:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758013.633509:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633510:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758013.633514:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:2.0:1787758013.633515:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758013.633519:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633520:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758013.633521:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633524:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787758013.633526:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758013.633527:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787758013.633528:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633530:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633532:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758013.633536:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758013.633536:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758013.633540:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758013.633541:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758013.633546:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633551:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758013.633552:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659705472:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758013.633559:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758013.633563:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:2.0:1787758013.633563:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659705600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758013.633566:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758013.633570:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758013.633576:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633578:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758013.633582:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633587:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758013.633590:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758013.633591:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758013.633595:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00080000:2.0:1787758013.633596:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758013.633600:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758013.633601:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787758013.633602:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758013.633606:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758013.633608:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11068 00000100:00080000:2.0:1787758013.633613:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758013.633615:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758013.633616:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758013.633618:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758013.633619:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758013.633621:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758013.633622:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758013.633624:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758013.633625:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758013.633628:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758013.633629:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758013.633630:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758013.633633:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000800:00000100:0.0:1787758013.633637:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:2.0:1787758013.633638:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633642:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758013.633644:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758013.633646:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633648:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633651:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787758013.633652:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758013.633656:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633658:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758013.633659:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758013.633663:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758013.633664:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758013.633665:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633667:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758013.633671:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633673:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633677:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305741505 00000100:00100000:2.0:1787758013.633680:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659705728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758013.633686:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758013.633693:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758013.633694:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305741505 00000100:00000001:2.0:1787758013.633697:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633701:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787758013.633703:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758013.633708:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758013.633711:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787758013.633713:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758013.633717:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633718:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758013.633720:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:2.0:1787758013.633723:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758013.633730:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758013.633732:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11068 00000100:00100000:0.0:1787758013.633734:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758013/real 1787758013] req@ffff8ffb47a38000 x1874588659705472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758013.633736:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758013.633740:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758013.633743:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758013.633747:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758013.633749:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000400:00000001:0.0:1787758013.633752:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:2.0:1787758013.633752:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758013.633754:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:0.0:1787758013.633756:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758013.633757:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758013.633760:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758013.633760:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758013.633764:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758013.633765:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758013.633768:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633769:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787758013.633771:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758013.633774:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:2.0:1787758013.633775:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758013.633777:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787758013.633779:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633782:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787758013.633784:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659705472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758013.633785:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758013.633790:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758013.633794:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758013.633799:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758013.633801:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787758013.633802:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633805:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758013.633805:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758013.633806:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758013.633810:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633811:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.633814:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758013.633814:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758013.633816:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633817:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758013.633821:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:2.0:1787758013.633825:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659705856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758013.633826:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758013.633835:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758013.633839:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758013.633840:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787758013.633843:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758013.633848:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659705472:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758013.633848:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633856:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.633858:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758013.633859:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758013.633862:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758013.633864:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758013.633865:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758013.633867:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758013.633870:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758013.633872:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758013.633875:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758013.633876:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.633877:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.633883:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758013.633884:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633886:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758013.633887:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758013.633888:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758013.633889:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758013.633890:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758013.633892:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758013.633893:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758013.633894:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758013.633896:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705600:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758013.633898:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758013.633899:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758013.633900:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758013.633901:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758013.633904:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633907:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758013.633908:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:0.0:1787758013.633911:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758013.633919:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758013.633922:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758013.633924:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758013.633925:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633926:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758013.633928:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633931:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305740813 00000100:00000001:0.0:1787758013.633933:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305741505 00000100:00000001:0.0:1787758013.633934:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305740813 00000100:00000001:0.0:1787758013.633936:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305741505 00000100:00000001:0.0:1787758013.633938:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758013.633939:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.633940:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758013.633943:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758013/real 1787758013] req@ffff8ffb47a38380 x1874588659705600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758013.633949:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758013.633950:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758013.633951:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758013.633952:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758013.633953:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.633954:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758013.633955:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758013.633958:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659705600/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758013.633962:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.633964:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758013.633965:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.633966:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758013.633967:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.633968:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758013.633977:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758013.633979:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758013.633981:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758013.633983:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758013.633985:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705600:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758013.633987:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758013.633988:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758013.633989:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758013.633990:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758013.633991:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758013.633992:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758013.633994:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758013.633998:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758013.633999:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758013.634000:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758013.634001:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758013.634003:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634006:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758013.634007:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787758013.634009:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758013.634015:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758013.634018:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758013.634019:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758013.634020:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634021:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758013.634023:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634025:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305740810 00000100:00000001:0.0:1787758013.634026:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305741505 00000100:00000001:0.0:1787758013.634028:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305740810 00000100:00000001:0.0:1787758013.634029:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305741505 00000100:00000001:0.0:1787758013.634031:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758013.634032:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634033:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758013.634035:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758013/real 1787758013] req@ffff8ffb47a38a80 x1874588659705728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758013.634040:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758013.634040:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758013.634042:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758013.634042:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758013.634043:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634044:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758013.634045:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758013.634047:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659705728/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758013.634056:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634057:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758013.634059:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634060:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758013.634061:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634062:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758013.634069:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758013.634071:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758013.634073:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758013.634074:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758013.634076:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758013.634078:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758013.634079:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758013.634080:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758013.634081:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758013.634082:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758013.634084:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758013.634085:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705856:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758013.634087:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758013.634088:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758013.634089:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758013.634090:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758013.634091:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634094:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758013.634095:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787758013.634097:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758013.634102:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758013.634105:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758013.634106:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758013.634107:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634108:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758013.634110:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634112:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305740810 00000100:00000001:0.0:1787758013.634114:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305741505 00000100:00000001:0.0:1787758013.634115:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305740810 00000100:00000001:0.0:1787758013.634117:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305741505 00000100:00000001:0.0:1787758013.634118:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758013.634119:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634120:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758013.634122:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758013/real 1787758013] req@ffff8ffb47a3a680 x1874588659705856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758013.634127:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758013.634127:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758013.634129:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758013.634129:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758013.634130:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634131:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758013.634132:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758013.634134:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659705856/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758013.634139:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634140:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758013.634142:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634143:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758013.634144:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634145:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758013.634153:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758013.634155:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758013.634156:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758013.634165:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758013.634168:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659705856:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758013.634173:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634174:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758013.634175:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758013.634176:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758013.634178:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758013.634178:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758013.634179:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758013.634208:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758013.634209:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758013.634210:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758013.634211:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634212:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758013.634213:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758013.634214:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758013.634216:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758013.634217:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758013.634218:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758013.634218:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758013.634219:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758013.634220:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758013.634221:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634222:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758013.634223:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758013.634223:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758013.634224:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758013.634225:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758013.634226:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758013.634227:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758013.634228:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758013.634230:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758013.634231:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634232:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758013.634234:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758013.634235:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758013.634236:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758013.634237:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758014.209394:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758014.209401:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758014.209403:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758014.209409:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758014.209412:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758014.209415:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758014.209417:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758014.209419:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758014.209422:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758014.401286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758014.401292:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758014.401296:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758014.401298:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758014.401344:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758014.401347:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758014.401350:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758015.233354:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758015.233363:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758015.233366:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758015.233373:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758015.233377:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758015.233382:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758015.233385:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758015.233389:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758015.233392:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758015.425297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758015.425330:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758015.425336:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758015.425343:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758015.425351:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758015.425356:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758015.425361:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758015.425367:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758015.425375:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758015.425378:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758015.425486:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758015.425492:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758015.425498:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758016.257369:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758016.257378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758016.257381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758016.257388:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758016.257392:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758016.257399:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758016.257402:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758016.257406:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758016.257409:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758016.449318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758016.449325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758016.449334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758016.449337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758016.449429:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758016.449438:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758016.449445:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758017.281244:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758017.281252:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758017.281254:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758017.281260:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758017.281264:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758017.281283:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758017.281286:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758017.281289:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758017.281292:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758017.473253:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758017.473260:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758017.473313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758017.473315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758017.473348:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758017.473351:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758017.473353:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758018.305258:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758018.305265:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758018.305267:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758018.305271:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758018.305274:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758018.305277:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758018.305279:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758018.305281:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758018.305283:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758018.497302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758018.497308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758018.497313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758018.497315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758018.497372:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758018.497376:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758018.497378:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758018.753343:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758018.753351:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758018.753353:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758018.753356:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758018.753360:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758018.753366:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11073 00000100:00080000:1.0:1787758018.753375:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758018.753378:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758018.753380:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758018.753383:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758018.753386:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758018.753388:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758018.753389:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758018.753391:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758018.753392:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758018.753396:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758018.753397:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758018.753399:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758018.753402:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758018.753404:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753406:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758018.753407:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758018.753418:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758018.753419:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753422:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758018.753424:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758018.753428:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753437:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758018.753444:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659705984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758018.753452:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758018.753455:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753468:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758018.753471:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758018.753473:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758018.753475:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758018.753476:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758018.753478:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758018.753479:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11073 00000100:00080000:1.0:1787758018.753481:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758018.753482:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758018.753483:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758018.753485:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758018.753486:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758018.753487:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758018.753489:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758018.753491:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758018.753492:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758018.753494:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758018.753495:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758018.753496:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758018.753500:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758018.753502:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753503:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758018.753507:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758018.753510:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753511:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758018.753514:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758018.753515:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753516:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758018.753517:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758018.753519:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753521:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758018.753522:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753526:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:1.0:1787758018.753528:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659706112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758018.753530:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758018.753534:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758018.753537:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753539:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758018.753542:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758018.753543:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758018.753545:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758018.753547:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758018.753548:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758018.753551:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758018.753553:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758018.753555:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758018.753558:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758018.753559:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659705984:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758018.753560:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758018.753565:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787758018.753566:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11073 00000100:00080000:1.0:1787758018.753570:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758018.753571:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758018.753572:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758018.753575:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753576:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758018.753580:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:1.0:1787758018.753581:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758018.753584:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758018.753586:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758018.753587:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787758018.753588:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753588:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758018.753592:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758018.753595:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758018.753596:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758018.753597:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758018.753600:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758018.753601:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753602:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758018.753604:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758018.753606:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00000001:1.0:1787758018.753606:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753610:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:1.0:1787758018.753610:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753616:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787758018.753617:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758018.753620:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753621:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758018.753622:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758018.753623:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753627:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758018.753630:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659706240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758018.753636:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758018.753637:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753639:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758018.753641:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758018.753643:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758018.753644:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758018.753645:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758018.753647:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758018.753649:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11073 00000100:00080000:1.0:1787758018.753652:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758018.753654:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758018.753655:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758018.753658:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758018.753660:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758018.753661:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758018.753662:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758018.753665:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758018.753666:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758018.753667:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758018.753668:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787758018.753672:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758018.753673:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758018.753674:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753678:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758018.753678:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758018.753683:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787758018.753683:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758018.753687:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753689:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758018.753690:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787758018.753690:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758018.753696:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753698:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305746625 00000100:00000001:1.0:1787758018.753698:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758018.753703:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758018.753705:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753706:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758018.753707:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758018.753708:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753709:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758018.753713:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753716:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305746625 00000100:00100000:1.0:1787758018.753719:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659706368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758018.753723:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758018.753726:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758018.753729:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758018.753730:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753739:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758018.753743:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753745:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758018.753751:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758018/real 1787758018] req@ffff8ffb4a535c00 x1874588659705984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758018.753760:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758018.753763:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758018.753766:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758018.753768:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758018.753770:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753772:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758018.753774:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758018.753778:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659705984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758018.753784:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753786:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758018.753788:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753789:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758018.753790:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753792:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758018.753793:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758018.753796:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758018.753798:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758018.753800:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758018.753802:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659705984:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758018.753805:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753807:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758018.753808:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758018.753809:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758018.753813:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758018.753814:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758018.753816:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758018.753819:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758018.753821:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758018.753822:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758018.753823:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753825:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753830:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758018.753832:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753833:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758018.753835:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758018.753836:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758018.753837:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758018.753838:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758018.753839:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758018.753840:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758018.753842:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758018.753844:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706112:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758018.753846:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758018.753847:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758018.753848:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758018.753849:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758018.753851:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753862:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758018.753863:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758018.753867:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758018.753875:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758018.753878:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758018.753880:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758018.753881:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753882:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758018.753884:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753887:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305745933 00000100:00000001:0.0:1787758018.753889:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305746625 00000100:00000001:0.0:1787758018.753890:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305745933 00000100:00000001:0.0:1787758018.753892:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305746625 00000100:00000001:0.0:1787758018.753893:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758018.753895:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753896:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758018.753898:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758018/real 1787758018] req@ffff8ffb4a537b80 x1874588659706112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758018.753903:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758018.753904:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758018.753905:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758018.753906:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758018.753907:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753909:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758018.753910:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758018.753912:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659706112/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758018.753916:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753917:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758018.753919:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753920:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758018.753921:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.753922:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758018.753933:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758018.753934:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758018.753936:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758018.753938:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758018.753941:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706112:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758018.753943:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758018.753944:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758018.753944:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758018.753946:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758018.753947:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758018.753948:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758018.753950:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758018.753952:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758018.753953:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758018.753954:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758018.753955:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758018.753957:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753960:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758018.753961:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758018.753963:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758018.753969:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758018.753971:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758018.753973:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758018.753974:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753975:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758018.753976:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753978:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305745930 00000100:00000001:0.0:1787758018.753980:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305746625 00000100:00000001:0.0:1787758018.753982:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305745930 00000100:00000001:0.0:1787758018.753983:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305746625 00000100:00000001:0.0:1787758018.753985:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758018.753986:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.753987:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758018.753990:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758018/real 1787758018] req@ffff8ffb4a537800 x1874588659706240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758018.753995:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758018.753995:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758018.753997:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758018.753998:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758018.753999:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754000:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758018.754001:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758018.754003:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659706240/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758018.754007:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754008:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758018.754009:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754010:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758018.754011:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754012:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758018.754021:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758018.754022:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758018.754024:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758018.754025:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758018.754028:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758018.754030:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758018.754031:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758018.754032:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758018.754033:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758018.754034:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758018.754035:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758018.754037:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758018.754039:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758018.754040:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758018.754041:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758018.754042:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758018.754044:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.754047:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758018.754047:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758018.754049:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758018.754055:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758018.754057:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758018.754058:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758018.754059:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.754060:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758018.754062:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.754064:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305745930 00000100:00000001:0.0:1787758018.754065:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305746625 00000100:00000001:0.0:1787758018.754067:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305745930 00000100:00000001:0.0:1787758018.754068:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305746625 00000100:00000001:0.0:1787758018.754069:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758018.754071:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.754072:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758018.754074:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758018/real 1787758018] req@ffff8ffb4a534700 x1874588659706368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758018.754078:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758018.754079:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758018.754080:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758018.754081:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758018.754082:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754083:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758018.754083:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758018.754085:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659706368/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758018.754094:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754095:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758018.754097:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754098:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758018.754099:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754100:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758018.754108:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758018.754109:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758018.754111:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758018.754112:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758018.754114:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706368:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758018.754116:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754117:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758018.754118:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758018.754119:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758018.754121:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758018.754122:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758018.754122:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758018.754124:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758018.754125:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758018.754126:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758018.754127:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754128:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758018.754129:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758018.754130:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758018.754131:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758018.754132:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758018.754133:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758018.754133:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758018.754134:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758018.754135:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758018.754136:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754137:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758018.754137:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758018.754138:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758018.754139:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758018.754140:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758018.754141:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758018.754143:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758018.754143:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758018.754145:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758018.754145:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754147:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758018.754148:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758018.754149:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758018.754150:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758018.754151:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758019.329249:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758019.329256:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758019.329258:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758019.329262:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758019.329265:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758019.329268:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758019.329270:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758019.329272:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758019.329274:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758019.521279:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758019.521287:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758019.521294:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758019.521296:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758019.521343:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758019.521347:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758019.521351:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758020.353263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758020.353269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758020.353271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758020.353275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758020.353278:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758020.353281:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758020.353283:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758020.353285:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758020.353286:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758020.545350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758020.545356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758020.545360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758020.545361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758020.545397:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758020.545400:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758020.545404:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758021.377274:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758021.377280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758021.377282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758021.377286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758021.377289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758021.377292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758021.377294:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758021.377295:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758021.377297:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758021.569288:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758021.569294:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758021.569299:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758021.569300:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758021.569316:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758021.569318:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758021.569320:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758022.401327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758022.401335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758022.401337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758022.401343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758022.401346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758022.401349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758022.401352:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758022.401354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758022.401357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758022.593269:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758022.593276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758022.593282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758022.593284:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758022.593330:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758022.593332:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758022.593335:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758023.425255:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758023.425261:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758023.425262:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758023.425267:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758023.425270:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758023.425274:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.425276:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758023.425278:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.425280:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758023.617244:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758023.617250:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758023.617254:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758023.617258:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758023.617261:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758023.617264:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758023.617267:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758023.617269:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758023.617271:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758023.617273:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758023.617317:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758023.617321:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758023.617323:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758023.873270:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758023.873279:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758023.873281:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758023.873283:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758023.873287:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758023.873292:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11078 00000100:00080000:3.0:1787758023.873303:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758023.873306:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758023.873308:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758023.873311:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758023.873313:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758023.873316:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758023.873317:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758023.873319:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758023.873320:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758023.873324:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758023.873326:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758023.873328:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758023.873331:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758023.873334:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873336:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758023.873338:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758023.873345:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758023.873347:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873349:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758023.873352:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758023.873355:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873364:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758023.873371:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659706496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758023.873380:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758023.873383:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873401:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758023.873405:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758023.873409:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758023.873411:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758023.873413:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758023.873416:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758023.873420:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11078 00000100:00080000:3.0:1787758023.873425:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758023.873428:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758023.873430:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758023.873433:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758023.873434:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758023.873437:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758023.873440:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758023.873442:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758023.873443:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758023.873447:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758023.873450:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758023.873450:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758023.873452:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758023.873454:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758023.873457:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758023.873458:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758023.873460:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758023.873461:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873463:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873463:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787758023.873467:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00100000:3.0:1787758023.873468:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758023.873470:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787758023.873472:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.873473:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873474:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758023.873479:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659706496:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787758023.873479:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758023.873481:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873484:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758023.873484:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758023.873486:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758023.873487:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.873488:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758023.873491:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.873494:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758023.873499:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873514:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758023.873517:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787758023.873521:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758023.873549:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758023.873557:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1787758023.873557:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659706624/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758023.873565:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758023.873567:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873568:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758023.873571:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758023.873571:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758023.873573:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758023.873574:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873576:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787758023.873577:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758023.873578:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758023.873579:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758023.873581:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873582:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758023.873585:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305751745 00000100:00080000:3.0:1787758023.873585:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11078 00000100:00080000:3.0:1787758023.873589:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758023.873591:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758023.873592:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758023.873593:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787758023.873594:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758023.873596:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758023.873597:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758023.873599:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305751745 00000020:00000001:3.0:1787758023.873600:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758023.873602:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758023.873602:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758023.873604:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758023.873606:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787758023.873607:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758023.873609:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873609:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758023.873611:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758023.873612:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787758023.873614:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758023.873616:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758023.873617:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758023/real 1787758023] req@ffff8ffb45c31500 x1874588659706496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758023.873618:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758023.873621:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758023.873624:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873626:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787758023.873628:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787758023.873629:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:1.0:1787758023.873630:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787758023.873631:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873633:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758023.873633:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758023.873634:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758023.873636:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787758023.873636:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873637:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873640:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758023.873640:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873642:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758023.873644:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659706752/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758023.873646:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659706496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758023.873652:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758023.873654:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873655:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758023.873656:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873658:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758023.873659:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758023.873661:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873663:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758023.873664:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873666:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758023.873668:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758023.873671:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758023.873675:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758023.873678:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758023.873683:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659706496:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758023.873688:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873691:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758023.873692:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758023.873695:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758023.873699:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758023.873701:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758023.873703:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758023.873706:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758023.873708:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758023.873710:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758023.873711:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873714:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873720:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758023.873721:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873724:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758023.873725:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758023.873727:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758023.873728:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758023.873729:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.873732:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758023.873733:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758023.873735:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758023.873739:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706624:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758023.873742:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758023.873744:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758023.873746:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.873748:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758023.873750:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873756:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758023.873758:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787758023.873763:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787758023.873764:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758023.873766:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758023.873768:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758023.873772:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000800:00000100:1.0:1787758023.873776:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758023.873781:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758023.873784:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787758023.873784:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11078 00000100:00000001:1.0:1787758023.873786:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873788:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787758023.873824:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758023.873826:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873827:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758023.873829:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758023.873831:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305751053 00000020:00000001:3.0:1787758023.873833:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758023.873835:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305751745 00000020:00000001:3.0:1787758023.873835:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758023.873837:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305751053 00000020:00000001:3.0:1787758023.873837:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758023.873840:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758023.873841:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305751745 00000100:00000001:1.0:1787758023.873843:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758023.873843:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758023.873845:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758023.873846:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873848:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:3.0:1787758023.873850:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1787758023.873853:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758023/real 1787758023] req@ffff8ffb45c32a00 x1874588659706624/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758023.873853:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758023.873855:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758023.873858:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758023.873860:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873862:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000400:00000001:1.0:1787758023.873863:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000008:00100000:3.0:1787758023.873864:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:1.0:1787758023.873865:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000008:00000001:3.0:1787758023.873867:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873868:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758023.873869:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873870:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758023.873872:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758023.873873:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758023.873874:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758023.873875:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758023.873875:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873877:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758023.873878:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787758023.873879:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659706624/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758023.873880:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873885:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873888:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873890:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:3.0:1787758023.873890:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659706880/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758023.873892:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873894:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758023.873896:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.873898:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758023.873903:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758023.873905:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758023.873907:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873911:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758023.873914:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758023.873917:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758023.873920:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758023.873924:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706624:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758023.873928:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758023.873929:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758023.873931:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.873933:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758023.873935:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758023.873937:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758023.873940:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758023.873943:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758023.873945:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758023.873947:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.873949:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758023.873952:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873958:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758023.873959:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787758023.873961:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758023.873971:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758023.873974:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758023.873976:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758023.873977:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873978:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758023.873980:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873982:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305751050 00000100:00000001:1.0:1787758023.873984:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305751745 00000100:00000001:1.0:1787758023.873985:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305751050 00000100:00000001:1.0:1787758023.873987:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305751745 00000100:00000001:1.0:1787758023.873988:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758023.873990:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.873991:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758023.873993:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758023/real 1787758023] req@ffff8ffb45c30700 x1874588659706752/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758023.873998:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758023.873999:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758023.874000:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758023.874002:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758023.874003:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874004:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758023.874005:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758023.874007:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659706752/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758023.874012:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874013:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758023.874015:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874016:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758023.874017:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874018:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758023.874026:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758023.874028:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758023.874030:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758023.874032:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758023.874034:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758023.874036:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874037:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758023.874038:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758023.874039:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758023.874040:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758023.874041:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758023.874042:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758023.874044:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758023.874045:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758023.874046:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758023.874047:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874048:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758023.874049:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758023.874050:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758023.874051:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758023.874052:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758023.874053:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758023.874054:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758023.874055:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758023.874055:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758023.874056:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874057:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874060:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758023.874061:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.874062:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758023.874063:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758023.874063:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758023.874064:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758023.874065:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.874067:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758023.874068:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758023.874069:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758023.874071:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706880:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758023.874073:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758023.874074:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758023.874075:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758023.874076:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758023.874078:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.874081:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758023.874082:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787758023.874084:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758023.874090:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758023.874093:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758023.874094:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758023.874095:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.874096:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758023.874098:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.874100:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305751050 00000100:00000001:1.0:1787758023.874101:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305751745 00000100:00000001:1.0:1787758023.874103:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305751050 00000100:00000001:1.0:1787758023.874104:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305751745 00000100:00000001:1.0:1787758023.874106:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758023.874107:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.874108:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758023.874111:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758023/real 1787758023] req@ffff8ffb45c30a80 x1874588659706880/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758023.874115:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758023.874116:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758023.874125:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758023.874126:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758023.874127:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874129:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758023.874130:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758023.874133:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659706880/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758023.874137:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874139:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758023.874140:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874141:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758023.874142:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874143:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758023.874178:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758023.874180:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758023.874208:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758023.874212:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758023.874217:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659706880:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758023.874222:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874224:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758023.874226:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758023.874227:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758023.874230:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758023.874231:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758023.874233:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758023.874235:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758023.874236:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758023.874238:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758023.874239:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874241:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758023.874243:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758023.874244:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758023.874245:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758023.874247:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758024.449245:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758024.449251:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758024.449253:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758024.449257:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758024.449259:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758024.449262:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758024.449264:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758024.449266:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758024.449268:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758024.641259:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758024.641264:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758024.641270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758024.641272:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758024.641302:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758024.641307:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758024.641312:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758025.473247:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758025.473255:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758025.473257:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758025.473263:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758025.473267:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758025.473270:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758025.473273:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758025.473276:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758025.473279:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758025.665244:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758025.665251:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758025.665253:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758025.665256:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758025.665261:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758025.665263:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758025.665265:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758025.665266:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758025.665269:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758025.665270:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758025.665287:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758025.665289:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758025.665290:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758026.497248:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758026.497254:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758026.497256:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758026.497260:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758026.497263:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758026.497266:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758026.497268:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758026.497270:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758026.497272:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758026.689245:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758026.689249:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758026.689254:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758026.689256:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758026.689307:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758026.689311:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758026.689313:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758027.521234:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758027.521240:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758027.521242:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758027.521246:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758027.521248:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758027.521251:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758027.521253:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758027.521255:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758027.521257:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758027.713263:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758027.713270:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758027.713275:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758027.713276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758027.713333:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758027.713338:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758027.713342:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758028.545256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758028.545263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758028.545265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758028.545269:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758028.545272:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758028.545275:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758028.545277:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758028.545279:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758028.545280:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758028.737262:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758028.737268:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758028.737273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758028.737274:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758028.737373:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758028.737379:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758028.737382:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758028.993272:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758028.993284:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758028.993286:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758028.993289:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758028.993294:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758028.993301:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11083 00000100:00080000:1.0:1787758028.993313:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758028.993316:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758028.993318:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758028.993322:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758028.993325:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758028.993327:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758028.993329:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758028.993331:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758028.993332:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758028.993336:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758028.993338:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758028.993340:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758028.993344:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758028.993346:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993348:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758028.993350:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758028.993361:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758028.993363:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993367:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758028.993370:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758028.993375:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993387:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758028.993398:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659707008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758028.993411:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758028.993415:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993434:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758028.993437:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758028.993440:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758028.993441:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758028.993442:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758028.993445:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758028.993447:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11083 00000100:00080000:1.0:1787758028.993449:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758028.993451:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758028.993452:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758028.993454:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758028.993455:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758028.993457:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758028.993458:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758028.993460:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758028.993461:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758028.993463:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758028.993465:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758028.993466:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758028.993472:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758028.993474:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993476:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758028.993481:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758028.993485:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993487:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758028.993492:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758028.993494:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993497:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758028.993499:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758028.993502:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993508:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.993514:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:1.0:1787758028.993515:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659707136/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758028.993522:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758028.993526:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758028.993530:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758028.993534:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993535:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758028.993542:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758028.993542:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758028.993548:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758028.993553:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993553:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758028.993569:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758028.993575:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787758028.993577:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659707008:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758028.993579:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758028.993582:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758028.993586:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787758028.993587:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787758028.993592:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:1.0:1787758028.993592:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11083 02000000:00000001:0.0:1787758028.993596:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758028.993602:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:1.0:1787758028.993605:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758028.993610:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758028.993611:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993613:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758028.993617:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758028.993618:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758028.993620:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758028.993622:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758028.993625:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758028.993626:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758028.993628:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758028.993633:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00080000:1.0:1787758028.993634:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758028.993641:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.993642:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787758028.993650:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758028.993655:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758028.993658:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993660:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758028.993662:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758028.993665:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993666:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758028.993671:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758028.993672:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993674:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758028.993675:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758028.993677:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993683:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758028.993687:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659707264/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758028.993698:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758028.993699:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758028.993705:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993709:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758028.993711:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758028.993715:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787758028.993715:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758028.993721:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758028.993725:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787758028.993725:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758028.993732:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993732:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758028.993740:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:1.0:1787758028.993744:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11083 00000100:00000001:0.0:1787758028.993751:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758028.993752:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758028.993761:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758028.993763:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305756865 00000100:00000001:1.0:1787758028.993764:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758028.993767:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758028.993768:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758028.993769:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758028.993779:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758028.993782:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758028.993783:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758028.993788:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758028.993788:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758028.993796:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758028.993799:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758028.993801:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305756865 02000000:00000001:1.0:1787758028.993802:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758028.993808:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.993810:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758028.993811:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758028.993816:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758028.993820:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993821:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758028.993824:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758028.993825:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.993826:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758028.993827:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758028.993834:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758028.993836:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758028.993843:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.993844:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758028.993856:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758028.993865:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758028/real 1787758028] req@ffff8ffb4a534a80 x1874588659707008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758028.993866:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659707392/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787758028.993886:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787758028.993889:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:0.0:1787758028.993891:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758028.993893:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758028.993897:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.993899:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758028.993906:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758028.993908:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.993912:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758028.993914:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758028.993925:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659707008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758028.993935:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.993939:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758028.993942:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.993944:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758028.993945:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.993948:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758028.993949:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758028.993952:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758028.993956:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758028.993958:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758028.993961:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659707008:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758028.993965:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.993967:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758028.993968:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758028.993970:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758028.993973:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758028.993975:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758028.993977:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758028.993981:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758028.993982:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758028.993984:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758028.993985:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.993987:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.993995:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758028.993997:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.993999:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758028.994001:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758028.994002:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758028.994004:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758028.994006:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758028.994008:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758028.994011:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758028.994013:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758028.994018:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707136:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758028.994021:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758028.994024:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758028.994025:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758028.994027:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758028.994031:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994037:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758028.994039:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:0.0:1787758028.994043:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758028.994055:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758028.994059:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758028.994075:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758028.994077:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994078:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758028.994081:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994085:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305756173 00000100:00000001:0.0:1787758028.994087:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305756865 00000100:00000001:0.0:1787758028.994089:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305756173 00000100:00000001:0.0:1787758028.994090:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305756865 00000100:00000001:0.0:1787758028.994092:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758028.994093:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994094:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758028.994097:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758028/real 1787758028] req@ffff8ffb4a536300 x1874588659707136/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758028.994103:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758028.994104:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758028.994106:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758028.994107:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758028.994108:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994109:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758028.994110:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758028.994113:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659707136/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758028.994122:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994124:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758028.994132:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994134:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758028.994136:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994138:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758028.994151:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758028.994154:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758028.994158:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758028.994160:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758028.994164:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707136:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758028.994168:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758028.994170:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758028.994172:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758028.994174:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758028.994176:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758028.994178:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758028.994211:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758028.994213:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758028.994215:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758028.994216:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758028.994218:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758028.994220:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994225:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758028.994226:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758028.994229:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758028.994238:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758028.994241:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758028.994243:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758028.994244:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994246:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758028.994247:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994250:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305756171 00000100:00000001:0.0:1787758028.994251:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305756865 00000100:00000001:0.0:1787758028.994253:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305756171 00000100:00000001:0.0:1787758028.994254:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305756865 00000100:00000001:0.0:1787758028.994256:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758028.994257:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994258:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758028.994261:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758028/real 1787758028] req@ffff8ffb4a534700 x1874588659707264/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758028.994266:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758028.994266:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758028.994268:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758028.994269:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758028.994270:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994271:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758028.994272:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758028.994274:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659707264/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758028.994278:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994280:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758028.994281:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994282:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758028.994283:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994284:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758028.994293:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758028.994294:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758028.994297:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758028.994298:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758028.994301:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758028.994303:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758028.994304:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758028.994305:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758028.994307:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758028.994308:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758028.994309:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758028.994311:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707392:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758028.994313:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758028.994315:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758028.994316:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758028.994317:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758028.994319:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994322:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758028.994323:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758028.994325:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758028.994331:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758028.994334:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758028.994335:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758028.994336:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994337:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758028.994338:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994341:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305756171 00000100:00000001:0.0:1787758028.994343:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305756865 00000100:00000001:0.0:1787758028.994344:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305756171 00000100:00000001:0.0:1787758028.994346:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305756865 00000100:00000001:0.0:1787758028.994347:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758028.994348:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994349:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758028.994351:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758028/real 1787758028] req@ffff8ffb4a537800 x1874588659707392/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758028.994360:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758028.994361:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758028.994363:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758028.994364:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758028.994364:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994365:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758028.994366:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758028.994368:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659707392/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758028.994372:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994373:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758028.994375:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994376:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758028.994377:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994378:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758028.994387:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758028.994389:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758028.994391:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758028.994393:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758028.994395:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707392:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758028.994397:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994398:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758028.994399:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758028.994400:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758028.994401:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758028.994402:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758028.994403:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758028.994405:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758028.994406:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758028.994407:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758028.994408:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994409:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758028.994410:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758028.994411:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758028.994413:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758028.994414:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758028.994414:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758028.994416:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758028.994417:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758028.994417:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758028.994418:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994419:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758028.994420:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758028.994420:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758028.994421:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758028.994422:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758028.994423:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758028.994423:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758028.994424:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758028.994425:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758028.994426:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994427:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758028.994429:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758028.994430:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758028.994431:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758028.994432:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758029.569275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758029.569284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758029.569285:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758029.569290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758029.569294:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758029.569298:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758029.569300:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758029.569302:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758029.569304:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758029.761332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758029.761339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758029.761345:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758029.761347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758029.761373:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758029.761377:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758029.761380:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758030.593275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758030.593286:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758030.593307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758030.593313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758030.593317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758030.593322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758030.593325:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758030.593329:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758030.593333:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758030.785300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758030.785309:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758030.785315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758030.785316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758030.785384:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758030.785388:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758030.785390:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758031.617268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758031.617276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758031.617278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758031.617283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758031.617287:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758031.617291:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758031.617294:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758031.617297:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758031.617299:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758031.809280:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758031.809285:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758031.809293:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758031.809295:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758031.809383:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758031.809386:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758031.809389:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758032.641277:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758032.641285:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758032.641287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758032.641293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758032.641297:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758032.641302:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758032.641305:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758032.641309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758032.641312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758032.833275:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758032.833284:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758032.833291:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758032.833293:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758032.833361:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758032.833365:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758032.833368:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758033.665297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758033.665305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758033.665308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758033.665313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758033.665316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758033.665321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758033.665323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758033.665326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758033.665329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758033.857332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758033.857340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758033.857346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758033.857350:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758033.857354:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758033.857358:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758033.857362:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758033.857364:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758033.857367:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758033.857369:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758033.857403:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758033.857406:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758033.857409:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758034.113283:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758034.113294:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758034.113297:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758034.113301:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758034.113305:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758034.113313:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11088 00000100:00080000:1.0:1787758034.113325:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758034.113328:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758034.113330:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758034.113335:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758034.113337:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758034.113340:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758034.113342:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758034.113344:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758034.113345:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758034.113348:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758034.113350:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113352:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758034.113355:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758034.113358:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113360:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758034.113363:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113377:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758034.113379:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113383:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758034.113387:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758034.113392:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113410:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758034.113422:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659707520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758034.113436:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758034.113441:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113460:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758034.113463:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758034.113466:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758034.113467:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758034.113468:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758034.113470:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758034.113472:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11088 00000100:00080000:1.0:1787758034.113476:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758034.113478:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758034.113479:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758034.113481:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758034.113482:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758034.113484:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758034.113485:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758034.113487:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758034.113487:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758034.113489:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758034.113491:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113492:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758034.113497:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758034.113498:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113500:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758034.113504:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758034.113506:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113508:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113512:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758034.113514:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113515:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758034.113516:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758034.113518:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113523:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758034.113530:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659707648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758034.113536:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758034.113538:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113540:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758034.113542:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758034.113545:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758034.113546:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758034.113552:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758034.113554:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758034.113555:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758034.113562:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758034.113563:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758034.113573:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:1.0:1787758034.113576:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11088 00000100:00000001:0.0:1787758034.113577:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758034.113581:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758034.113587:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758034.113588:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758034.113593:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758034.113594:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787758034.113602:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758034.113604:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758034.113606:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787758034.113610:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758034.113613:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00100000:0.0:1787758034.113621:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659707520:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758034.113621:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758034.113628:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758034.113631:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758034.113636:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758034.113642:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758034.113642:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787758034.113655:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758034.113656:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113663:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758034.113667:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758034.113669:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113670:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113683:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758034.113686:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758034.113690:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113691:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113696:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758034.113697:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787758034.113697:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.113703:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:1.0:1787758034.113703:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758034.113709:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758034.113710:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787758034.113714:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113721:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758034.113727:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659707776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758034.113733:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758034.113735:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113737:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758034.113740:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758034.113743:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758034.113745:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758034.113746:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758034.113749:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758034.113751:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11088 00000800:00000100:0.0:1787758034.113755:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787758034.113755:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758034.113769:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758034.113772:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758034.113774:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758034.113776:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758034.113777:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758034.113780:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:1.0:1787758034.113780:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758034.113785:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758034.113787:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758034.113788:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758034.113794:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758034.113794:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758034.113805:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758034.113805:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113815:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758034.113819:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758034.113820:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758034.113830:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.113831:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305761985 00000100:00000001:1.0:1787758034.113834:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758034.113836:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758034.113839:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113841:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.113843:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:1.0:1787758034.113845:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758034.113848:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113849:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758034.113852:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305761985 02000000:00000001:1.0:1787758034.113852:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758034.113856:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.113861:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758034.113863:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.113866:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758034.113873:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758034.113873:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659707904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758034.113882:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758034.113891:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:0.0:1787758034.113893:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758034/real 1787758034] req@ffff8ffb4a537b80 x1874588659707520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758034.113894:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.113904:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758034.113917:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758034.113920:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758034.113924:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758034.113927:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758034.113929:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.113933:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758034.113935:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758034.113941:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659707520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758034.113951:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.113954:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758034.113966:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.113968:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758034.113970:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114042:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758034.114044:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758034.114047:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758034.114051:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758034.114052:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758034.114055:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659707520:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758034.114061:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758034.114063:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758034.114064:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758034.114066:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758034.114068:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758034.114069:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758034.114071:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707648:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758034.114074:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758034.114075:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758034.114077:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758034.114078:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758034.114081:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114086:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758034.114087:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:0.0:1787758034.114091:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758034.114101:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758034.114105:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758034.114107:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758034.114108:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114109:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758034.114111:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114114:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305761293 00000100:00000001:0.0:1787758034.114116:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305761985 00000100:00000001:0.0:1787758034.114117:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305761293 00000100:00000001:0.0:1787758034.114119:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305761985 00000100:00000001:0.0:1787758034.114120:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758034.114121:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114123:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758034.114125:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758034/real 1787758034] req@ffff8ffb4a535c00 x1874588659707648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758034.114131:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758034.114132:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758034.114134:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758034.114135:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758034.114136:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114137:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758034.114138:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758034.114140:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659707648/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758034.114145:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114146:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758034.114148:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114149:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758034.114150:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114151:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758034.114161:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758034.114162:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758034.114164:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758034.114166:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758034.114168:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707648:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758034.114171:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114173:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758034.114174:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758034.114176:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758034.114179:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758034.114210:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758034.114212:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758034.114215:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758034.114217:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758034.114219:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758034.114220:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114221:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758034.114222:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758034.114223:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758034.114224:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758034.114225:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758034.114226:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758034.114228:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758034.114229:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758034.114230:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758034.114231:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114232:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114238:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758034.114239:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114241:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758034.114242:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758034.114243:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758034.114244:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758034.114245:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758034.114247:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758034.114248:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758034.114250:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758034.114252:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758034.114254:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758034.114256:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758034.114257:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758034.114258:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758034.114260:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114264:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758034.114265:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758034.114268:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758034.114275:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758034.114278:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758034.114280:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758034.114281:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114282:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758034.114284:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114286:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305761291 00000100:00000001:0.0:1787758034.114288:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305761985 00000100:00000001:0.0:1787758034.114289:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305761291 00000100:00000001:0.0:1787758034.114291:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305761985 00000100:00000001:0.0:1787758034.114292:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758034.114294:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114295:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758034.114298:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758034/real 1787758034] req@ffff8ffb4a537800 x1874588659707776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758034.114303:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758034.114304:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758034.114305:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758034.114306:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758034.114307:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114308:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758034.114309:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758034.114311:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659707776/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758034.114316:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114317:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758034.114319:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114320:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758034.114321:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114322:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758034.114330:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758034.114332:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758034.114333:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758034.114335:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758034.114337:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758034.114339:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758034.114341:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758034.114342:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758034.114343:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758034.114344:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758034.114345:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758034.114347:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707904:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758034.114349:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758034.114350:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758034.114351:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758034.114353:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758034.114354:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114357:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758034.114358:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758034.114360:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758034.114366:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758034.114368:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758034.114369:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758034.114370:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114372:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758034.114373:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114375:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305761291 00000100:00000001:0.0:1787758034.114377:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305761985 00000100:00000001:0.0:1787758034.114378:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305761291 00000100:00000001:0.0:1787758034.114380:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305761985 00000100:00000001:0.0:1787758034.114381:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758034.114382:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114383:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758034.114386:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758034/real 1787758034] req@ffff8ffb4a534700 x1874588659707904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758034.114390:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758034.114391:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758034.114392:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758034.114393:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758034.114393:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114394:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758034.114395:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758034.114397:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659707904/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758034.114406:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114408:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758034.114409:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114410:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758034.114411:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114412:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758034.114420:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758034.114422:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758034.114423:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758034.114425:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758034.114426:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659707904:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758034.114429:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114430:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758034.114431:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758034.114432:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758034.114433:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758034.114434:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758034.114435:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758034.114436:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758034.114437:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758034.114438:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758034.114439:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114440:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758034.114441:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758034.114442:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758034.114443:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758034.114444:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758034.114444:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758034.114445:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758034.114446:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758034.114447:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758034.114448:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114449:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758034.114450:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758034.114452:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758034.114453:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758034.114454:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758034.689246:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758034.689253:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758034.689256:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758034.689261:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758034.689265:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758034.689268:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758034.689271:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758034.689273:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758034.689276:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758034.881285:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758034.881292:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758034.881300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758034.881302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758034.881349:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758034.881353:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758034.881357:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758035.713292:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758035.713301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758035.713303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758035.713308:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758035.713311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758035.713315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758035.713317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758035.713320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758035.713322:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758035.905257:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758035.905263:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758035.905266:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758035.905269:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758035.905274:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758035.905275:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758035.905277:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758035.905280:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758035.905283:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758035.905284:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758035.905332:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758035.905335:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758035.905338:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758036.737272:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758036.737280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758036.737282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758036.737287:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758036.737290:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758036.737294:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758036.737296:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758036.737299:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758036.737302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758036.929255:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758036.929260:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758036.929273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758036.929275:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758036.929336:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758036.929339:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758036.929342:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758037.761298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758037.761306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758037.761309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758037.761313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758037.761317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758037.761321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758037.761324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758037.761326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758037.761329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758037.953297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758037.953305:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758037.953311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758037.953312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758037.953334:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758037.953347:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758037.953350:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758038.785232:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758038.785241:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758038.785243:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758038.785248:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758038.785252:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758038.785255:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758038.785257:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758038.785259:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758038.785262:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758038.977321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758038.977330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758038.977338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758038.977340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758038.977475:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758038.977481:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758038.977486:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758039.233402:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758039.233414:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758039.233418:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758039.233422:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758039.233427:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758039.233437:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11093 00000100:00080000:3.0:1787758039.233451:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758039.233455:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758039.233458:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758039.233463:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758039.233466:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758039.233470:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758039.233473:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758039.233476:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758039.233477:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758039.233483:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758039.233486:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758039.233490:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758039.233494:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758039.233498:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233501:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758039.233503:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758039.233517:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758039.233519:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233523:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758039.233526:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758039.233532:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233546:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758039.233559:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659708032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758039.233574:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758039.233580:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233606:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758039.233611:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758039.233615:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758039.233617:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758039.233619:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758039.233623:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758039.233627:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11093 00000100:00080000:3.0:1787758039.233631:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758039.233634:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758039.233636:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758039.233639:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758039.233640:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758039.233643:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758039.233645:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758039.233648:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758039.233649:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758039.233653:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758039.233656:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758039.233658:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758039.233666:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758039.233669:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233671:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758039.233678:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758039.233682:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233685:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758039.233689:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758039.233691:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233693:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758039.233695:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758039.233699:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233708:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758039.233720:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659708160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758039.233737:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758039.233738:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758039.233746:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758039.233749:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.233752:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758039.233760:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758039.233764:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233765:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758039.233770:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787758039.233776:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758039.233777:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758039.233787:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758039.233795:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787758039.233801:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659708032:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758039.233804:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758039.233812:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758039.233816:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758039.233825:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758039.233830:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758039.233836:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758039.233840:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787758039.233847:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758039.233853:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11093 00000100:00080000:3.0:1787758039.233867:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758039.233871:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758039.233872:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758039.233877:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:3.0:1787758039.233878:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758039.233906:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000020:00000001:3.0:1787758039.233909:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758039.233916:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758039.233921:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787758039.233922:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758039.233929:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758039.233931:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758039.233937:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758039.233941:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758039.233943:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758039.233949:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758039.233952:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233955:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758039.233958:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758039.233962:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233965:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758039.233972:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758039.233974:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.233977:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758039.233979:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758039.233983:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758039.233991:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758039.233996:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758039.234008:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659708288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758039.234009:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758039.234018:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758039.234024:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.234027:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758039.234030:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758039.234034:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234041:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.234043:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234054:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305767105 00000100:00080000:3.0:1787758039.234055:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758039.234067:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758039.234068:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758039.234071:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758039.234074:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758039.234075:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758039.234084:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11093 00000100:00000001:0.0:1787758039.234088:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305767105 00000100:00080000:3.0:1787758039.234092:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758039.234098:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758039.234099:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758039.234101:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758039.234108:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758039.234109:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758039.234114:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758039.234118:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758039.234120:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758039.234126:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:3.0:1787758039.234128:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758039.234134:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758039.234140:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787758039.234142:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758039/real 1787758039] req@ffff8ffb45c30e00 x1874588659708032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758039.234149:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758039.234161:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758039.234170:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000400:00000001:0.0:1787758039.234178:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787758039.234179:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:0.0:1787758039.234223:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758039.234230:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758039.234235:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758039.234239:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758039.234242:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234245:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758039.234248:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758039.234252:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:3.0:1787758039.234254:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758039.234265:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.234270:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758039.234271:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659708032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758039.234278:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758039.234283:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.234290:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758039.234294:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758039.234300:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.234312:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234317:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234325:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:3.0:1787758039.234327:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659708416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758039.234332:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234340:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758039.234344:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234353:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758039.234359:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758039.234361:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00080000:0.0:1787758039.234368:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787758039.234370:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758039.234379:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758039.234386:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758039.234392:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758039.234398:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659708032:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758039.234405:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234409:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758039.234412:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758039.234416:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758039.234422:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758039.234425:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758039.234427:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758039.234433:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758039.234435:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758039.234438:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758039.234440:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234443:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234452:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758039.234454:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234457:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758039.234459:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758039.234461:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758039.234463:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758039.234466:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758039.234469:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758039.234471:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758039.234474:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758039.234479:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708160:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758039.234483:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758039.234486:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758039.234488:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758039.234491:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758039.234495:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234505:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758039.234509:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:0.0:1787758039.234517:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758039.234542:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758039.234552:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758039.234557:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758039.234560:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234563:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758039.234568:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234587:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305766414 00000100:00000001:0.0:1787758039.234593:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305767105 00000100:00000001:0.0:1787758039.234676:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305766414 00000100:00000001:0.0:1787758039.234682:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305767105 00000100:00000001:0.0:1787758039.234687:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758039.234692:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234696:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758039.234708:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758039/real 1787758039] req@ffff8ffb45c31c00 x1874588659708160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758039.234726:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758039.234728:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758039.234734:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758039.234737:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758039.234739:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234741:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758039.234743:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758039.234749:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659708160/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758039.234767:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234770:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758039.234773:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234775:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758039.234777:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234779:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758039.234794:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758039.234797:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758039.234801:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758039.234804:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758039.234808:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708160:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758039.234829:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758039.234831:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758039.234833:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758039.234836:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758039.234839:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758039.234842:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758039.234846:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758039.234850:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758039.234853:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758039.234855:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758039.234858:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758039.234862:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234870:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758039.234872:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:0.0:1787758039.234878:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758039.234928:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758039.234936:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758039.234939:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758039.234941:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234944:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758039.234947:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234952:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305766411 00000100:00000001:0.0:1787758039.234955:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305767105 00000100:00000001:0.0:1787758039.234958:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305766411 00000100:00000001:0.0:1787758039.234961:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305767105 00000100:00000001:0.0:1787758039.234964:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758039.234966:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.234968:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758039.234974:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758039/real 1787758039] req@ffff8ffb45c33b80 x1874588659708288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758039.234983:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758039.234985:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758039.234988:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758039.234990:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758039.234992:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.234994:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758039.234996:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758039.235001:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659708288/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758039.235009:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235012:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758039.235015:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235017:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758039.235019:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235021:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758039.235033:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758039.235036:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758039.235040:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758039.235042:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758039.235047:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758039.235052:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758039.235054:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758039.235056:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758039.235058:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758039.235060:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758039.235063:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758039.235066:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758039.235070:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758039.235072:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758039.235074:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758039.235077:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758039.235080:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.235086:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758039.235088:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787758039.235093:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758039.235106:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758039.235111:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758039.235114:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758039.235115:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.235118:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758039.235120:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.235124:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305766411 00000100:00000001:0.0:1787758039.235128:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305767105 00000100:00000001:0.0:1787758039.235130:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305766411 00000100:00000001:0.0:1787758039.235133:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305767105 00000100:00000001:0.0:1787758039.235136:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758039.235138:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.235140:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758039.235145:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758039/real 1787758039] req@ffff8ffb45c30a80 x1874588659708416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758039.235162:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758039.235163:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758039.235166:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758039.235168:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758039.235169:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235171:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758039.235173:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758039.235177:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659708416/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758039.235219:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235222:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758039.235225:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235227:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758039.235228:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235230:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758039.235243:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758039.235246:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758039.235250:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758039.235263:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758039.235267:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708416:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758039.235272:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235275:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758039.235277:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758039.235279:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758039.235283:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758039.235285:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758039.235287:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758039.235290:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758039.235292:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758039.235295:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758039.235297:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235299:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758039.235301:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758039.235302:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758039.235304:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758039.235306:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758039.235307:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758039.235309:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758039.235311:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758039.235313:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758039.235314:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235316:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758039.235317:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758039.235319:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758039.235321:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758039.235322:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758039.235323:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758039.235325:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758039.235326:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758039.235328:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758039.235329:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235332:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758039.235337:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758039.235338:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758039.235341:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758039.235343:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758039.809331:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758039.809344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758039.809347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758039.809355:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758039.809360:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758039.809366:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758039.809370:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758039.809375:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758039.809379:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758040.001428:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758040.001441:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758040.001455:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758040.001458:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758040.001540:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758040.001548:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758040.001554:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758040.833291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758040.833300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758040.833302:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758040.833308:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758040.833312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758040.833316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758040.833318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758040.833321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758040.833323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758041.025326:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758041.025352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758041.025360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758041.025362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758041.025494:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758041.025500:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758041.025505:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758041.857335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758041.857352:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758041.857356:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758041.857368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758041.857376:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758041.857384:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758041.857389:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758041.857395:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758041.857401:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758042.049348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758042.049357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758042.049363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758042.049366:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758042.049392:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758042.049395:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758042.049399:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758042.881320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758042.881332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758042.881335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758042.881342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758042.881346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758042.881351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758042.881354:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758042.881357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758042.881360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758043.073255:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758043.073263:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758043.073270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758043.073271:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758043.073347:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758043.073353:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758043.073358:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758043.905303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758043.905313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758043.905316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758043.905322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758043.905327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758043.905332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758043.905335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758043.905338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758043.905342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758044.097336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758044.097344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758044.097350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758044.097355:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758044.097358:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758044.097363:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758044.097367:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758044.097369:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758044.097372:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758044.097375:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758044.097484:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758044.097491:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758044.097496:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758044.353359:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758044.353375:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758044.353378:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758044.353384:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758044.353391:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758044.353402:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11099 00000100:00080000:3.0:1787758044.353419:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758044.353423:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758044.353426:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758044.353432:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758044.353437:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758044.353443:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758044.353466:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758044.353469:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758044.353471:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758044.353477:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758044.353481:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758044.353485:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758044.353491:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758044.353496:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353499:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758044.353514:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758044.353533:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758044.353535:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353539:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758044.353542:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758044.353548:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353563:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758044.353574:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659708544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758044.353585:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758044.353590:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353608:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758044.353612:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758044.353615:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758044.353617:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758044.353619:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758044.353622:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758044.353624:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11099 00000100:00080000:3.0:1787758044.353628:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758044.353630:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758044.353632:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758044.353634:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758044.353635:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758044.353637:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758044.353639:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758044.353641:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758044.353642:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758044.353645:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758044.353647:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758044.353649:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758044.353654:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758044.353657:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353659:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758044.353664:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758044.353666:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353668:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758044.353700:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758044.353702:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353704:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758044.353706:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758044.353709:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353717:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758044.353724:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659708672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758044.353734:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758044.353736:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353739:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758044.353745:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758044.353746:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787758044.353750:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758044.353757:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758044.353760:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787758044.353763:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758044.353773:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758044.353780:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11099 00000100:00000001:0.0:1787758044.353783:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:3.0:1787758044.353789:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758044.353812:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758044.353820:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758044.353821:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353828:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758044.353837:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:3.0:1787758044.353839:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758044.353846:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787758044.353847:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758044.353853:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787758044.353858:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758044.353864:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758044.353870:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758044.353874:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787758044.353882:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659708544:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787758044.353884:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758044.353898:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758044.353909:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758044.353910:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758044.353919:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758044.353923:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758044.353930:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758044.353931:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758044.353937:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758044.353942:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787758044.353950:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787758044.353953:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758044.353964:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353969:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.353976:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787758044.353979:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758044.353980:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 02000000:00000001:3.0:1787758044.353984:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.353990:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758044.353992:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787758044.354006:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758044.354012:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.354022:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758044.354032:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659708800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758044.354043:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758044.354046:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.354052:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758044.354056:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758044.354062:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787758044.354072:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758044.354077:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758044.354080:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758044.354088:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758044.354090:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787758044.354095:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11099 00000100:00000001:0.0:1787758044.354101:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787758044.354105:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758044.354107:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.354111:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758044.354113:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758044.354120:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758044.354125:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758044.354126:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758044.354138:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758044.354141:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305772225 00000020:00000001:3.0:1787758044.354145:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758044.354153:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758044.354158:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758044.354161:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758044.354162:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787758044.354172:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758044.354178:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354231:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305772225 02000000:00000001:3.0:1787758044.354241:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758044.354243:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758044.354251:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758044.354254:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758044.354257:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354274:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787758044.354275:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.354284:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787758044.354288:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758044/real 1787758044] req@ffff8ffb45c30700 x1874588659708544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758044.354291:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758044.354312:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758044.354341:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787758044.354342:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758044.354350:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787758044.354357:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758044.354358:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787758044.354366:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354369:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758044.354373:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758044.354375:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758044.354378:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758044.354382:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787758044.354386:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354388:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758044.354414:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758044.354416:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659708544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758044.354444:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758044.354448:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659708928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758044.354451:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758044.354458:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354460:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758044.354462:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354471:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758044.354475:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758044.354486:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758044.354506:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758044.354519:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787758044.354522:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:0.0:1787758044.354528:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659708544:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758044.354529:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758044.354544:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354548:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758044.354552:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758044.354556:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758044.354561:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758044.354565:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758044.354568:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758044.354575:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708672:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758044.354580:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758044.354583:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758044.354585:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758044.354587:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758044.354592:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354599:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758044.354601:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787758044.354607:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758044.354625:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758044.354631:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758044.354634:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758044.354636:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354638:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758044.354642:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354647:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305771534 00000100:00000001:0.0:1787758044.354651:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305772225 00000100:00000001:0.0:1787758044.354654:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305771534 00000100:00000001:0.0:1787758044.354657:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305772225 00000100:00000001:0.0:1787758044.354660:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758044.354662:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354664:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758044.354670:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758044/real 1787758044] req@ffff8ffb45c32a00 x1874588659708672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758044.354679:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758044.354681:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758044.354683:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758044.354686:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758044.354687:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354690:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758044.354692:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758044.354697:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659708672/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758044.354706:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354708:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758044.354711:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354713:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758044.354715:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354717:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758044.354733:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758044.354736:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758044.354740:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758044.354743:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758044.354746:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708672:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758044.354751:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354754:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758044.354756:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758044.354759:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758044.354764:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758044.354769:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758044.354773:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758044.354779:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758044.354782:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758044.354788:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758044.354813:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354818:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758044.354821:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758044.354823:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758044.354829:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758044.354831:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758044.354834:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758044.354840:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758044.354842:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758044.354848:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758044.354852:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354857:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.354870:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758044.354875:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354881:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758044.354884:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758044.354887:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758044.354891:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758044.354893:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758044.354899:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758044.354901:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758044.354904:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758044.354910:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758044.354915:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758044.354918:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758044.354920:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758044.354922:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758044.354927:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354936:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758044.354937:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787758044.354943:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758044.354961:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758044.354967:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758044.354970:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758044.354972:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354974:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758044.354978:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354983:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305771531 00000100:00000001:0.0:1787758044.354986:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305772225 00000100:00000001:0.0:1787758044.354989:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305771531 00000100:00000001:0.0:1787758044.354991:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305772225 00000100:00000001:0.0:1787758044.354994:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758044.354996:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.354999:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758044.355004:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758044/real 1787758044] req@ffff8ffb45c31500 x1874588659708800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758044.355015:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758044.355017:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758044.355020:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758044.355022:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758044.355023:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355026:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758044.355028:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758044.355032:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659708800/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758044.355041:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355044:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758044.355046:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355048:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758044.355050:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355052:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758044.355065:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758044.355068:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758044.355071:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758044.355074:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758044.355077:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758044.355082:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758044.355084:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758044.355085:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758044.355088:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758044.355090:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758044.355093:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758044.355096:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758044.355101:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758044.355103:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758044.355106:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758044.355109:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758044.355112:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.355119:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758044.355121:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787758044.355126:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758044.355139:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758044.355144:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758044.355147:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758044.355149:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.355151:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758044.355154:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.355158:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305771531 00000100:00000001:0.0:1787758044.355161:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305772225 00000100:00000001:0.0:1787758044.355164:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305771531 00000100:00000001:0.0:1787758044.355167:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305772225 00000100:00000001:0.0:1787758044.355170:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758044.355172:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.355174:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758044.355178:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758044/real 1787758044] req@ffff8ffb45c30000 x1874588659708928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758044.355223:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758044.355224:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758044.355227:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758044.355229:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758044.355231:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355233:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758044.355235:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758044.355240:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659708928/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758044.355249:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355252:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758044.355255:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355257:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758044.355259:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355261:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758044.355275:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758044.355278:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758044.355282:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758044.355285:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758044.355289:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659708928:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758044.355293:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355295:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758044.355297:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758044.355299:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758044.355301:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758044.355304:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758044.355305:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758044.355308:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758044.355322:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758044.355324:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758044.355326:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355328:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758044.355330:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758044.355331:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758044.355333:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758044.355335:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758044.355336:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758044.355338:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758044.355339:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758044.355341:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758044.355343:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355345:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758044.355349:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758044.355351:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758044.355353:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758044.355355:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758044.929331:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758044.929344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758044.929347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758044.929357:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758044.929363:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758044.929368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758044.929372:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758044.929376:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758044.929380:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758045.121346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758045.121357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758045.121366:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758045.121370:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758045.121422:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758045.121432:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758045.121438:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758045.953269:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758045.953276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758045.953278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758045.953282:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758045.953286:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758045.953289:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758045.953291:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758045.953293:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758045.953296:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758046.145337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758046.145347:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758046.145351:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758046.145356:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758046.145363:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758046.145366:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758046.145369:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758046.145372:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758046.145376:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758046.145378:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758046.145406:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758046.145409:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758046.145412:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758046.977351:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758046.977360:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758046.977363:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758046.977370:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758046.977374:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758046.977380:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758046.977383:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758046.977386:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758046.977389:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758047.169333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758047.169348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758047.169373:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758047.169375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758047.169465:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758047.169475:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758047.169483:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758048.001351:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758048.001363:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758048.001366:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758048.001374:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758048.001380:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758048.001403:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758048.001407:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758048.001411:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758048.001416:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758048.193307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758048.193318:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758048.193325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758048.193327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758048.193463:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758048.193471:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758048.193478:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758049.025327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758049.025344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758049.025348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758049.025359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758049.025366:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758049.025374:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758049.025380:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758049.025386:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758049.025392:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758049.217344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758049.217353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758049.217360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758049.217362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758049.217468:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758049.217475:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758049.217483:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758049.473276:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758049.473288:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758049.473292:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758049.473297:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758049.473303:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758049.473313:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11104 00000100:00080000:3.0:1787758049.473326:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758049.473330:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758049.473333:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758049.473338:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758049.473341:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758049.473345:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758049.473349:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758049.473351:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758049.473353:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758049.473359:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758049.473363:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758049.473366:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758049.473370:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758049.473375:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473377:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758049.473381:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758049.473393:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758049.473395:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473399:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758049.473402:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758049.473407:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473422:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758049.473435:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659709056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758049.473449:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758049.473455:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473476:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758049.473480:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758049.473488:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758049.473490:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758049.473492:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758049.473496:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758049.473499:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11104 00000100:00080000:3.0:1787758049.473503:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758049.473506:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758049.473508:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758049.473513:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758049.473517:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758049.473522:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758049.473528:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758049.473533:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758049.473536:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758049.473545:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787758049.473546:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758049.473555:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758049.473557:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.473561:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758049.473569:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758049.473569:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758049.473578:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.473586:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758049.473586:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758049.473593:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758049.473597:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.473600:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473605:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787758049.473614:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659709056:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1787758049.473618:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758049.473625:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:3.0:1787758049.473630:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.473634:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758049.473639:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.473641:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.473652:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758049.473658:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758049.473661:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758049.473666:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.473683:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758049.473688:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:3.0:1787758049.473693:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758049.473696:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787758049.473723:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758049.473733:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473745:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758049.473757:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659709184/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758049.473783:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758049.473786:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758049.473793:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473800:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.473802:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787758049.473805:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758049.473811:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787758049.473813:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758049.473818:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473819:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758049.473828:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758049.473829:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758049.473838:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473842:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758049.473846:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305777345 00000100:00080000:3.0:1787758049.473856:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11104 00000100:00000001:0.0:1787758049.473864:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787758049.473870:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758049.473875:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305777345 00000100:00000001:3.0:1787758049.473878:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758049.473887:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758049.473887:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758049.473898:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758049.473900:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758049.473903:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758049.473908:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758049.473911:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:3.0:1787758049.473919:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00100000:0.0:1787758049.473926:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758049/real 1787758049] req@ffff8ffb45c33480 x1874588659709056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787758049.473928:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758049.473940:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758049.473945:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758049.473958:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787758049.473960:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758049.473966:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758049.473968:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.473973:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787758049.473974:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758049.473981:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787758049.473984:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758049.473986:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758049.473994:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.473997:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758049.474001:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758049.474004:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.474013:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474015:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758049.474019:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758049.474022:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758049.474029:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758049.474031:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659709056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758049.474034:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758049.474040:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758049.474046:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474058:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758049.474060:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474064:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758049.474072:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758049.474074:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659709312/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758049.474076:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758049.474081:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474090:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758049.474095:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758049.474104:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00080000:0.0:1787758049.474106:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787758049.474114:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758049.474119:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758049.474127:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787758049.474128:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758049.474135:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659709056:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787758049.474140:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758049.474144:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474152:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787758049.474160:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758049.474161:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758049.474168:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787758049.474171:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787758049.474174:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00080000:3.0:1787758049.474179:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000001:0.0:1787758049.474182:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:3.0:1787758049.474243:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000020:00000001:0.0:1787758049.474251:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00080000:3.0:1787758049.474253:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11104 02000000:00000001:0.0:1787758049.474265:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00080000:3.0:1787758049.474266:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758049.474277:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758049.474277:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758049.474286:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758049.474291:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000020:00000001:3.0:1787758049.474292:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758049.474298:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758049.474301:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787758049.474303:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758049.474310:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787758049.474311:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758049.474319:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758049.474323:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758049.474326:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:3.0:1787758049.474329:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758049.474332:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.474336:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474339:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787758049.474343:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758049.474345:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758049.474350:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758049.474352:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758049.474354:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758049.474359:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758049.474362:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.474366:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758049.474368:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787758049.474373:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:3.0:1787758049.474375:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787758049.474379:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787758049.474384:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.474392:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758049.474393:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709184:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758049.474402:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758049.474403:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758049.474406:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758049.474408:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.474412:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.474414:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758049.474421:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758049.474422:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758049.474430:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.474432:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.474441:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474449:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758049.474456:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00100000:3.0:1787758049.474457:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659709440/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758049.474465:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758049.474476:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758049.474480:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758049.474487:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758049.474497:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758049.474507:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758049.474513:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758049.474515:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474518:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758049.474523:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474528:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305776654 00000100:00000001:0.0:1787758049.474532:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305777345 00000100:00000001:0.0:1787758049.474535:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305776654 00000100:00000001:0.0:1787758049.474538:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305777345 00000100:00000001:0.0:1787758049.474541:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758049.474543:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474546:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758049.474551:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758049/real 1787758049] req@ffff8ffb45c31180 x1874588659709184/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758049.474563:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758049.474565:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758049.474567:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758049.474569:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758049.474571:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474573:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758049.474575:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758049.474580:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659709184/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758049.474589:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474591:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758049.474594:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474596:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758049.474598:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474601:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758049.474616:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758049.474619:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758049.474624:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758049.474627:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758049.474631:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709184:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758049.474635:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758049.474637:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758049.474639:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.474641:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758049.474643:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758049.474646:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758049.474649:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758049.474653:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758049.474655:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758049.474657:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.474660:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758049.474663:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474669:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758049.474670:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787758049.474675:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758049.474687:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758049.474692:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758049.474695:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758049.474697:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474699:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758049.474732:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474738:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305776651 00000100:00000001:0.0:1787758049.474742:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305777345 00000100:00000001:0.0:1787758049.474745:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305776651 00000100:00000001:0.0:1787758049.474748:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305777345 00000100:00000001:0.0:1787758049.474751:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758049.474753:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474756:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758049.474762:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758049/real 1787758049] req@ffff8ffb45c33800 x1874588659709312/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758049.474772:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758049.474774:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758049.474778:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758049.474780:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758049.474782:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474784:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758049.474786:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758049.474790:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659709312/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758049.474799:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474802:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758049.474804:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474806:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758049.474808:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474810:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758049.474823:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758049.474826:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758049.474830:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758049.474833:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758049.474837:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758049.474842:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474844:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758049.474846:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758049.474848:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758049.474850:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758049.474852:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758049.474854:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758049.474858:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758049.474859:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758049.474862:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758049.474864:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474866:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758049.474867:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758049.474869:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758049.474872:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758049.474873:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758049.474875:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758049.474877:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758049.474878:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758049.474880:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758049.474881:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474883:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.474887:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758049.474889:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474892:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758049.474893:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758049.474895:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758049.474896:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758049.474898:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.474901:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758049.474903:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758049.474905:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758049.474909:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758049.474913:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758049.474916:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758049.474918:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758049.474920:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758049.474924:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474930:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758049.474932:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:0.0:1787758049.474936:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758049.474952:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758049.474958:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758049.474960:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758049.474962:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474965:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758049.474968:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474971:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305776651 00000100:00000001:0.0:1787758049.474975:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305777345 00000100:00000001:0.0:1787758049.474977:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305776651 00000100:00000001:0.0:1787758049.474980:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305777345 00000100:00000001:0.0:1787758049.474983:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758049.474985:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.474987:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758049.474992:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758049/real 1787758049] req@ffff8ffb45c33480 x1874588659709440/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758049.475001:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758049.475002:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758049.475005:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758049.475006:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758049.475008:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.475010:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758049.475012:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758049.475017:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659709440/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758049.475025:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.475027:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758049.475030:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.475032:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758049.475033:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.475035:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758049.475048:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758049.475051:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758049.475055:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758049.475058:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758049.475063:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709440:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758049.475067:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.475069:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758049.475071:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758049.475073:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758049.475075:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758049.475076:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758049.475078:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758049.475080:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758049.475081:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758049.475083:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758049.475085:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.475087:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758049.475090:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758049.475092:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758049.475095:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758049.475097:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758050.049331:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758050.049344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758050.049348:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758050.049356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758050.049362:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758050.049368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758050.049372:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758050.049376:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758050.049380:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758050.241391:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758050.241401:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758050.241410:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758050.241412:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758050.241441:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758050.241444:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758050.241448:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758051.073324:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758051.073337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758051.073340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758051.073347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758051.073353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758051.073359:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758051.073362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758051.073368:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758051.073373:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758051.265375:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758051.265395:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758051.265401:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758051.265403:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758051.265493:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758051.265506:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758051.265532:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758052.097364:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758052.097378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758052.097382:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758052.097390:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758052.097395:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758052.097401:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758052.097404:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758052.097408:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758052.097412:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758052.289345:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758052.289353:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758052.289360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758052.289362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758052.289494:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758052.289500:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758052.289505:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758053.121366:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758053.121382:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758053.121387:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758053.121398:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758053.121405:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758053.121413:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758053.121419:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758053.121425:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758053.121430:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758053.313344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758053.313351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758053.313357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758053.313361:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758053.313364:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758053.313367:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758053.313371:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758053.313374:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758053.313376:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758053.313378:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758053.313445:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758053.313450:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758053.313454:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758054.145314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758054.145326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758054.145330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758054.145337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758054.145344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758054.145350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758054.145354:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758054.145359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758054.145363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758054.337317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758054.337340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758054.337347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758054.337350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758054.337382:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758054.337386:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758054.337389:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758054.593326:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758054.593339:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758054.593343:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758054.593348:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758054.593354:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758054.593363:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11109 00000100:00080000:1.0:1787758054.593376:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758054.593380:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758054.593383:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758054.593388:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758054.593391:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758054.593396:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758054.593399:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758054.593401:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758054.593403:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758054.593409:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758054.593413:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758054.593416:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758054.593420:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758054.593424:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593427:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758054.593429:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758054.593442:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758054.593445:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593449:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758054.593453:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758054.593458:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593472:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758054.593484:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659709568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758054.593498:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758054.593503:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593523:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758054.593528:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758054.593532:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758054.593534:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758054.593536:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758054.593540:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758054.593543:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11109 00000100:00080000:1.0:1787758054.593548:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758054.593551:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758054.593553:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758054.593556:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758054.593558:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758054.593561:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758054.593565:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758054.593569:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758054.593572:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758054.593578:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758054.593584:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758054.593588:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758054.593597:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758054.593602:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593606:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758054.593634:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758054.593641:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00000001:1.0:1787758054.593643:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593668:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.593669:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758054.593677:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758054.593678:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758054.593682:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787758054.593685:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.593688:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593692:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758054.593697:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758054.593699:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758054.593702:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758054.593707:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758054.593708:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593724:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758054.593725:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659709568:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758054.593736:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:1.0:1787758054.593741:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659709696/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758054.593744:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758054.593750:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758054.593758:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758054.593766:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758054.593768:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593770:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593781:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758054.593785:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758054.593789:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758054.593792:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758054.593794:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787758054.593795:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758054.593800:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:1.0:1787758054.593802:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758054.593811:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787758054.593812:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11109 00000100:00080000:1.0:1787758054.593821:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758054.593824:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758054.593826:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758054.593830:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758054.593832:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758054.593835:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758054.593837:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758054.593840:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758054.593842:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758054.593847:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758054.593851:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758054.593853:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758054.593857:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758054.593859:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593862:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758054.593866:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758054.593869:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:1.0:1787758054.593871:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593878:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758054.593886:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758054.593888:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593890:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758054.593892:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758054.593894:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787758054.593896:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593905:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.593907:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00100000:1.0:1787758054.593915:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659709824/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758054.593916:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.593928:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758054.593936:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758054.593940:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.593941:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.593946:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.593950:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305782465 00000100:00080000:1.0:1787758054.593951:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758054.593961:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758054.593963:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758054.593965:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758054.593968:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758054.593970:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758054.593972:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11109 00000100:00080000:1.0:1787758054.593982:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758054.593983:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305782465 00000100:00000001:1.0:1787758054.593985:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758054.593989:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758054.593995:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:1.0:1787758054.593996:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758054.593999:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758054.594007:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758054.594013:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758054.594016:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758054.594018:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758054.594025:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758054.594026:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758054.594033:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594034:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594043:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:1.0:1787758054.594044:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758054.594053:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787758054.594057:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758054/real 1787758054] req@ffff8ffb4a536300 x1874588659709568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787758054.594061:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.594076:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758054.594084:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000400:00000001:0.0:1787758054.594091:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000008:00000001:1.0:1787758054.594093:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758054.594095:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758054.594103:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594104:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758054.594111:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758054.594114:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758054.594115:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758054.594118:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:1.0:1787758054.594120:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594123:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758054.594126:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758054.594130:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:0.0:1787758054.594134:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659709568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787758054.594146:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758054.594157:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594158:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594169:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1787758054.594174:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659709952/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758054.594176:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758054.594191:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758054.594226:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758054.594231:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594234:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758054.594237:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758054.594252:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758054.594253:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787758054.594266:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758054.594267:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758054.594277:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758054.594284:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659709568:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758054.594302:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594305:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758054.594309:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758054.594312:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758054.594318:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758054.594321:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758054.594323:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758054.594328:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758054.594330:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758054.594333:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758054.594335:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594338:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594346:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758054.594348:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594351:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758054.594353:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758054.594355:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758054.594357:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758054.594359:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758054.594362:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758054.594364:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758054.594367:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758054.594371:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709696:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758054.594375:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758054.594387:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758054.594389:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758054.594391:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758054.594395:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594402:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758054.594404:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:0.0:1787758054.594409:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758054.594426:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758054.594433:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758054.594436:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758054.594437:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594449:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758054.594453:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594458:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305781774 00000100:00000001:0.0:1787758054.594461:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305782465 00000100:00000001:0.0:1787758054.594464:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305781774 00000100:00000001:0.0:1787758054.594467:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305782465 00000100:00000001:0.0:1787758054.594470:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758054.594472:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594475:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758054.594480:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758054/real 1787758054] req@ffff8ffb4a534a80 x1874588659709696/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758054.594491:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758054.594492:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758054.594495:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758054.594497:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758054.594498:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594500:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758054.594502:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758054.594506:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659709696/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758054.594522:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594524:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758054.594527:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594529:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758054.594530:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594532:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758054.594545:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758054.594548:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758054.594553:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758054.594556:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758054.594563:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709696:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758054.594570:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758054.594572:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758054.594575:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758054.594579:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758054.594582:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758054.594585:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758054.594592:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709824:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758054.594598:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758054.594601:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758054.594604:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758054.594607:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758054.594636:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594648:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758054.594651:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758054.594667:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758054.594688:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758054.594694:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758054.594697:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758054.594699:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594701:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758054.594705:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594709:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305781771 00000100:00000001:0.0:1787758054.594712:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305782465 00000100:00000001:0.0:1787758054.594714:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305781771 00000100:00000001:0.0:1787758054.594717:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305782465 00000100:00000001:0.0:1787758054.594720:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758054.594722:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594724:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758054.594729:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758054/real 1787758054] req@ffff8ffb4a534700 x1874588659709824/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758054.594738:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758054.594740:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758054.594742:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758054.594744:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758054.594746:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594748:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758054.594750:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758054.594755:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659709824/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758054.594771:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594774:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758054.594776:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594778:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758054.594780:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594781:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758054.594794:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758054.594796:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758054.594800:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758054.594802:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758054.594806:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709824:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758054.594811:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758054.594812:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758054.594814:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758054.594817:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758054.594819:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758054.594821:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758054.594825:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709952:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758054.594829:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758054.594831:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758054.594833:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758054.594835:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758054.594838:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594846:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758054.594849:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758054.594855:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758054.594873:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758054.594881:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758054.594885:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758054.594888:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594892:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758054.594897:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594903:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305781771 00000100:00000001:0.0:1787758054.594907:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305782465 00000100:00000001:0.0:1787758054.594912:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305781771 00000100:00000001:0.0:1787758054.594916:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305782465 00000100:00000001:0.0:1787758054.594920:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758054.594924:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.594927:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758054.594935:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758054/real 1787758054] req@ffff8ffb4a537800 x1874588659709952/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758054.594948:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758054.594950:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758054.594954:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758054.594956:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758054.594957:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594959:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758054.594961:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758054.594966:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659709952/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758054.594974:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594977:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758054.594979:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594981:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758054.594982:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.594984:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758054.594996:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758054.594999:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758054.595002:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758054.595004:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758054.595018:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659709952:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758054.595023:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.595025:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758054.595027:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758054.595029:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758054.595032:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758054.595034:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758054.595035:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758054.595038:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758054.595040:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758054.595042:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758054.595044:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.595046:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758054.595047:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758054.595049:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758054.595051:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758054.595052:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758054.595053:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758054.595055:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758054.595056:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758054.595058:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758054.595059:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.595061:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758054.595062:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758054.595063:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758054.595065:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758054.595066:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758054.595067:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758054.595068:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758054.595069:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758054.595082:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758054.595084:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.595086:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758054.595090:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758054.595092:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758054.595094:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758054.595096:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758055.169272:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758055.169283:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758055.169286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758055.169293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758055.169298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758055.169303:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758055.169307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758055.169310:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758055.169314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758055.361268:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758055.361274:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758055.361281:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758055.361285:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758055.361338:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758055.361343:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758055.361346:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758056.193239:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758056.193246:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758056.193248:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758056.193255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758056.193258:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758056.193263:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758056.193265:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758056.193268:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758056.193271:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758056.385315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758056.385323:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758056.385327:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758056.385331:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758056.385335:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758056.385337:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758056.385339:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758056.385341:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758056.385344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758056.385345:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758056.385407:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758056.385413:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758056.385417:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758057.217327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758057.217342:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758057.217346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758057.217356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758057.217363:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758057.217384:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758057.217388:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758057.217391:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758057.217396:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758057.409285:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758057.409290:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758057.409294:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758057.409296:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758057.409369:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758057.409372:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758057.409375:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758058.241296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758058.241307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758058.241310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758058.241317:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758058.241322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758058.241327:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758058.241332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758058.241337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758058.241341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758058.433250:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758058.433255:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758058.433260:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758058.433262:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758058.433299:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758058.433302:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758058.433304:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758059.265247:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758059.265253:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758059.265255:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758059.265259:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758059.265262:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758059.265265:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758059.265267:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758059.265268:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758059.265271:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758059.457374:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758059.457385:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758059.457398:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758059.457401:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758059.457583:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758059.457593:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758059.457599:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758059.713441:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758059.713455:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758059.713461:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758059.713467:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758059.713477:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758059.713489:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11114 00000100:00080000:2.0:1787758059.713508:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758059.713514:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758059.713517:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758059.713550:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758059.713554:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758059.713559:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758059.713564:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758059.713568:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758059.713570:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758059.713578:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758059.713581:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758059.713585:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758059.713591:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758059.713597:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.713600:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758059.713604:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758059.713619:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758059.713622:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.713627:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758059.713632:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758059.713640:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.713665:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758059.713685:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659710080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758059.713702:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758059.713710:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.713733:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758059.713741:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758059.713751:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758059.713754:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758059.713756:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758059.713762:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758059.713767:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11114 00000100:00080000:2.0:1787758059.713772:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758059.713775:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758059.713778:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758059.713778:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758059.713787:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758059.713791:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758059.713792:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787758059.713797:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758059.713800:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:2.0:1787758059.713803:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758059.713807:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758059.713809:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758059.713814:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.713815:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758059.713828:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787758059.713830:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787758059.713833:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.713838:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758059.713839:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758059.713848:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758059.713851:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758059.713857:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758059.713861:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659710080:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758059.713862:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758059.713872:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758059.713874:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:2.0:1787758059.713881:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.713885:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787758059.713886:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.713895:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758059.713898:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758059.713902:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758059.713903:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.713909:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758059.713911:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758059.713913:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758059.713914:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.713929:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.713948:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:2.0:1787758059.713950:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659710208/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758059.713955:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:0.0:1787758059.713969:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758059.713981:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758059.713985:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.713990:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758059.713994:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758059.713999:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758059.714001:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758059.714003:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758059.714008:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758059.714014:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11114 00000100:00080000:2.0:1787758059.714022:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758059.714025:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758059.714028:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758059.714031:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758059.714034:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758059.714037:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758059.714041:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758059.714043:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758059.714045:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758059.714049:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758059.714053:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714054:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758059.714058:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758059.714065:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305787585 02000000:00000001:2.0:1787758059.714066:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758059.714078:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.714081:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758059.714089:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758059.714091:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00000001:2.0:1787758059.714098:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.714104:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714111:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305787585 02000000:00000001:2.0:1787758059.714115:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758059.714120:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.714124:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758059.714127:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787758059.714128:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758059.714134:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714138:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758059.714144:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.714145:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714152:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714160:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758059.714162:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659710336/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758059.714175:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758059.714176:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758059.714238:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.714244:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758059.714248:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758059.714252:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758059.714254:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758059.714257:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758059.714261:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758059.714266:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11114 00000100:00000001:0.0:1787758059.714270:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00080000:2.0:1787758059.714272:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758059.714284:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758059.714289:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758059.714290:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:2.0:1787758059.714295:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758059.714299:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758059.714300:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787758059.714303:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758059.714310:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:2.0:1787758059.714310:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758059.714319:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758059.714321:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758059.714324:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758059.714331:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758059.714333:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758059.714344:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.714348:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758059.714350:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758059.714358:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758059.714360:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758059.714373:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.714374:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.714377:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758059.714385:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758059.714393:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758059.714394:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710208:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758059.714398:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758059.714410:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758059.714412:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758059.714416:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758059.714422:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758059.714423:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758059.714427:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758059.714432:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758059.714434:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.714445:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758059.714450:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.714458:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758059.714460:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659710464/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758059.714474:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758059.714479:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758059.714480:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714488:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:2.0:1787758059.714490:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714508:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758059.714582:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714595:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305786894 00000100:00000001:0.0:1787758059.714604:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305787585 00000100:00000001:0.0:1787758059.714610:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305786894 00000100:00000001:0.0:1787758059.714615:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305787585 00000100:00000001:0.0:1787758059.714619:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758059.714624:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714645:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758059.714649:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758059.714652:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.714657:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758059.714667:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758059.714671:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758059.714681:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758059.714688:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758059.714692:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758059.714695:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.714698:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758059.714706:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714718:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758059.714721:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787758059.714728:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758059.714753:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714761:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305786891 00000100:00000001:0.0:1787758059.714766:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305787585 00000100:00000001:0.0:1787758059.714770:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305786891 00000100:00000001:0.0:1787758059.714775:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305787585 00000100:00000001:0.0:1787758059.714779:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758059.714782:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714787:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714791:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758059.714798:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758059.714802:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787758059.714806:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758059.714809:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758059.714811:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758059.714814:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758059.714816:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.714821:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758059.714825:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758059.714828:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758059.714836:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710464:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758059.714842:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758059.714846:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758059.714848:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758059.714852:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758059.714857:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714868:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758059.714871:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:0.0:1787758059.714877:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758059.714899:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714906:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305786891 00000100:00000001:0.0:1787758059.714911:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305787585 00000100:00000001:0.0:1787758059.714914:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305786891 00000100:00000001:0.0:1787758059.714918:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305787585 00000100:00000001:0.0:1787758059.714922:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758059.714926:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714931:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714934:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758059.714939:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758059.714941:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787758059.714945:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758059.714947:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758059.714950:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758059.714955:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758060.289298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758060.289310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758060.289314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758060.289322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758060.289327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758060.289331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758060.289335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758060.289338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758060.289342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758060.481368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758060.481380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758060.481391:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758060.481394:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758060.481433:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758060.481442:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758060.481448:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758061.313298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758061.313305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758061.313308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758061.313313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758061.313316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758061.313320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758061.313323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758061.313326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758061.313329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758061.505348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758061.505357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758061.505365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758061.505367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758061.505539:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758061.505544:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758061.505548:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758062.337312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758062.337320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758062.337323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758062.337330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758062.337335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758062.337340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758062.337343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758062.337347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758062.337351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758062.529321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758062.529330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758062.529338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758062.529358:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758062.529388:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758062.529392:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758062.529397:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787758062.785585:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1787758062.785637:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787758062.785658:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758062.785674:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758062.785695:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758062.785699:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758062.785701:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787758062.785711:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758062.785716:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758062.785719:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758062.785721:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758062.785723:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787758062.785728:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758062.785733:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758062.785736:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758062.785738:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758062.785740:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787758062.785747:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758062.785757:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758062.785762:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758062.785764:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758062.785768:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758062.785777:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758062.785787:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758062.785792:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758062.785811:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758059/real 1787758062] req@ffff8ffb47a38a80 x1874588659710464/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758062.785856:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758062.785860:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758062.785879:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758062.785885:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758062.785888:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.785893:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758062.785896:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758062.785906:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659710464/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758062.785922:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.785928:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758062.785934:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.785937:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758062.785940:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.785944:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758062.785966:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758062.785973:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758062.785983:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758062.785988:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758062.785995:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710464:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758062.786013:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758062.786021:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758059/real 1787758062] req@ffff8ffb47a39500 x1874588659710208/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758062.786037:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758062.786040:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758062.786046:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758062.786049:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758062.786051:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786054:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758062.786057:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758062.786064:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659710208/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758062.786075:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786079:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758062.786083:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786085:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758062.786088:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786090:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758062.786110:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758062.786114:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758062.786121:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758062.786125:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758062.786130:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710208:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758062.786138:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758062.786145:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758059/real 1787758062] req@ffff8ffb47a3a680 x1874588659710336/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758062.786160:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758062.786162:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758062.786166:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758062.786170:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758062.786172:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786175:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758062.786178:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758062.786235:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659710336/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758062.786250:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786254:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758062.786259:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786263:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758062.786265:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786269:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758062.786288:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758062.786294:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758062.786300:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758062.786305:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758062.786311:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758062.786319:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758062.786326:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758059/real 1787758062] req@ffff8ffb47a39880 x1874588659710080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758062.786342:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758062.786344:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758062.786350:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758062.786353:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758062.786356:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786361:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758062.786364:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758062.786371:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659710080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758062.786384:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786387:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758062.786392:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786395:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758062.786397:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786400:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758062.786403:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758062.786408:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758062.786415:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758062.786420:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758062.786427:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659710080:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758062.786436:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786442:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758062.786445:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758062.786450:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758062.786459:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758062.786464:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758062.786490:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758062.786498:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758062.786502:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758062.786508:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758062.786510:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786515:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758062.786518:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758062.786521:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758062.786525:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758062.786528:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758062.786531:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758062.786536:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758062.786538:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758062.786544:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758062.786546:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786551:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758062.786554:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758062.786556:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758062.786560:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758062.786562:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758062.786565:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758062.786568:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758062.786570:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758062.786574:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758062.786576:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786580:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758062.786582:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758062.786584:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758062.786588:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758062.786590:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758062.786592:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758062.786596:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758062.786598:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758062.786601:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758062.786603:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786608:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758062.786620:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758062.786623:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758062.786627:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758062.786630:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758063.361290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758063.361298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758063.361301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758063.361306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758063.361310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758063.361314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758063.361318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758063.361321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758063.361324:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758063.553268:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758063.553274:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758063.553279:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758063.553282:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758063.553285:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758063.553287:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758063.553290:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758063.553291:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758063.553293:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758063.553294:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758063.553335:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758063.553339:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758063.553341:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758064.385284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758064.385293:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758064.385296:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758064.385302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758064.385306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758064.385311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758064.385315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758064.385317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758064.385320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758064.577428:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758064.577464:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758064.577504:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758064.577506:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758064.577577:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758064.577581:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758064.577585:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758064.833375:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758064.833386:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758064.833388:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758064.833392:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758064.833397:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758064.833405:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11119 00000100:00080000:1.0:1787758064.833416:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758064.833420:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758064.833422:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758064.833426:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758064.833429:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758064.833464:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758064.833467:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758064.833470:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758064.833471:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758064.833476:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758064.833479:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758064.833482:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758064.833486:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758064.833489:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833491:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758064.833494:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758064.833503:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758064.833505:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833508:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758064.833511:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758064.833516:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833528:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758064.833538:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659710592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758064.833549:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758064.833553:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833581:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758064.833586:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758064.833591:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758064.833594:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758064.833597:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758064.833601:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758064.833605:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11119 00000100:00080000:1.0:1787758064.833611:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758064.833614:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758064.833616:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758064.833620:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758064.833622:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758064.833625:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758064.833628:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758064.833631:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758064.833634:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758064.833640:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758064.833644:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758064.833647:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758064.833655:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758064.833659:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833663:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758064.833669:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758064.833674:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833678:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758064.833689:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758064.833690:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787758064.833692:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833702:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758064.833705:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758064.833706:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787758064.833710:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.833717:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758064.833726:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758064.833726:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.833735:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758064.833745:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659710720/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758064.833752:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758064.833763:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758064.833774:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758064.833776:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833785:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833796:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758064.833798:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659710592:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787758064.833805:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758064.833819:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787758064.833822:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758064.833830:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758064.833833:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787758064.833834:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758064.833840:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787758064.833843:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758064.833856:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11119 02000000:00000001:0.0:1787758064.833857:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:1.0:1787758064.833868:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758064.833872:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758064.833874:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758064.833877:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758064.833879:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758064.833882:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758064.833887:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787758064.833889:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833893:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758064.833901:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758064.833908:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758064.833912:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758064.833915:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758064.833920:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758064.833924:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.833928:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758064.833928:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758064.833936:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000008:00100000:1.0:1787758064.833939:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758064.833951:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00000001:1.0:1787758064.833952:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833961:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758064.833972:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758064.833974:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833977:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758064.833980:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758064.833982:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.833992:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758064.834003:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659710848/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758064.834036:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000800:00000100:0.0:1787758064.834047:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758064.834060:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.834069:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834074:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787758064.834075:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758064.834087:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758064.834090:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758064.834092:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758064.834098:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758064.834100:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758064.834111:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758064.834114:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758064.834127:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11119 00000100:00000001:0.0:1787758064.834129:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758064.834137:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758064.834142:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305792705 00000100:00000001:1.0:1787758064.834146:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758064.834153:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758064.834159:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758064.834162:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758064.834166:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758064.834168:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:1.0:1787758064.834171:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758064.834220:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758064.834224:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758064.834231:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758064.834235:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758064.834238:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758064.834244:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758064.834247:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.834249:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758064.834252:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758064.834258:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.834262:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758064.834270:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758064.834273:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.834275:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758064.834277:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758064.834282:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834283:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305792706 00000100:00000001:1.0:1787758064.834292:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834302:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00100000:1.0:1787758064.834309:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659710976/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758064.834325:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758064.834327:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758064.834331:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834342:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758064.834352:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834357:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758064.834373:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758064/real 1787758064] req@ffff8ffb4a535c00 x1874588659710592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758064.834397:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758064.834401:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758064.834409:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758064.834415:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758064.834420:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834425:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758064.834429:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758064.834467:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659710592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758064.834484:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834491:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758064.834498:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834501:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758064.834504:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834510:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758064.834514:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758064.834521:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758064.834531:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758064.834536:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758064.834546:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659710592:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758064.834557:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834562:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758064.834565:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758064.834570:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758064.834580:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758064.834587:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758064.834592:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758064.834601:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758064.834605:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758064.834611:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758064.834613:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834619:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834633:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758064.834637:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834642:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758064.834645:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758064.834648:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758064.834651:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758064.834654:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758064.834659:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758064.834664:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758064.834668:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758064.834675:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710720:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758064.834682:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758064.834687:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758064.834690:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758064.834693:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758064.834699:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834711:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758064.834714:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758064.834723:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758064.834752:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758064.834764:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758064.834769:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758064.834772:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834775:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758064.834780:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834789:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305792014 00000100:00000001:0.0:1787758064.834796:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305792705 00000100:00000001:0.0:1787758064.834801:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305792014 00000100:00000001:0.0:1787758064.834806:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305792706 00000100:00000001:0.0:1787758064.834810:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758064.834815:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.834819:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758064.834829:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758064/real 1787758064] req@ffff8ffb4a537b80 x1874588659710720/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758064.834847:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758064.834850:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758064.834855:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758064.834859:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758064.834862:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834866:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758064.834870:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758064.834878:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659710720/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758064.834894:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834898:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758064.834904:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834907:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758064.834910:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.834913:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758064.834942:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758064.834948:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758064.834955:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758064.834960:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758064.834968:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710720:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758064.834977:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758064.834980:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758064.834983:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758064.834988:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758064.834992:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758064.834996:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758064.835003:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758064.835009:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758064.835014:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758064.835017:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758064.835021:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758064.835029:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835039:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758064.835043:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758064.835051:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758064.835083:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758064.835093:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758064.835099:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758064.835102:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835106:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758064.835112:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835119:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305792011 00000100:00000001:0.0:1787758064.835124:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305792705 00000100:00000001:0.0:1787758064.835128:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305792011 00000100:00000001:0.0:1787758064.835132:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305792706 00000100:00000001:0.0:1787758064.835136:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758064.835141:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835144:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758064.835153:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758064/real 1787758064] req@ffff8ffb4a537800 x1874588659710848/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758064.835171:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758064.835174:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758064.835179:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758064.835230:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758064.835234:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835238:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758064.835241:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758064.835250:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659710848/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758064.835266:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835271:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758064.835276:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835279:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758064.835282:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835286:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758064.835322:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758064.835328:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758064.835335:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758064.835341:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758064.835348:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758064.835355:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758064.835359:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758064.835362:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758064.835367:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758064.835370:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758064.835375:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758064.835381:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710976:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758064.835387:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758064.835391:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758064.835394:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758064.835397:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758064.835404:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835415:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758064.835418:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758064.835425:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758064.835477:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758064.835486:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758064.835491:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758064.835494:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835497:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758064.835502:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835509:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305792012 00000100:00000001:0.0:1787758064.835514:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305792705 00000100:00000001:0.0:1787758064.835519:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305792012 00000100:00000001:0.0:1787758064.835523:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305792706 00000100:00000001:0.0:1787758064.835526:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758064.835530:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835534:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758064.835543:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758064/real 1787758064] req@ffff8ffb4a534700 x1874588659710976/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758064.835572:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758064.835574:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758064.835580:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758064.835584:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758064.835587:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835591:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758064.835594:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758064.835603:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659710976/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758064.835616:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835621:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758064.835626:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835629:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758064.835632:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835636:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758064.835660:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758064.835666:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758064.835673:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758064.835677:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758064.835684:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659710976:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758064.835692:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835696:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758064.835699:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758064.835703:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758064.835709:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758064.835712:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758064.835714:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758064.835721:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758064.835724:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758064.835728:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758064.835731:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835737:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758064.835738:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758064.835740:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758064.835742:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758064.835744:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758064.835746:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758064.835748:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758064.835749:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758064.835751:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758064.835752:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835754:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758064.835755:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758064.835757:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758064.835759:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758064.835760:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758064.835761:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758064.835765:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758064.835766:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758064.835768:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758064.835770:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835772:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758064.835778:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758064.835780:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758064.835783:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758064.835786:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758065.409254:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758065.409262:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758065.409265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758065.409272:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758065.409276:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758065.409280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758065.409283:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758065.409285:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758065.409288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758065.601360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758065.601372:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758065.601398:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758065.601400:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758065.601442:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758065.601447:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758065.601451:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758066.433253:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758066.433259:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758066.433261:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758066.433265:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758066.433269:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758066.433272:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758066.433274:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758066.433275:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758066.433277:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758066.625260:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758066.625268:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758066.625270:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758066.625273:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758066.625277:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758066.625279:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758066.625281:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758066.625283:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758066.625286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758066.625287:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758066.625306:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758066.625308:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758066.625310:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758067.457317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758067.457330:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758067.457335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758067.457344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758067.457351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758067.457357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758067.457360:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758067.457363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758067.457366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758067.649336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758067.649346:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758067.649356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758067.649360:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758067.649437:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758067.649444:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758067.649449:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758068.481313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758068.481327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758068.481331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758068.481339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758068.481344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758068.481350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758068.481354:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758068.481358:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758068.481362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758068.673272:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758068.673278:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758068.673284:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758068.673285:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758068.673390:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758068.673396:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758068.673418:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758069.505318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758069.505331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758069.505334:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758069.505343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758069.505390:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758069.505397:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758069.505401:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758069.505405:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758069.505410:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758069.697312:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758069.697325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758069.697336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758069.697339:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758069.697461:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758069.697467:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758069.697472:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758069.953317:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758069.953333:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758069.953337:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758069.953362:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758069.953370:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758069.953382:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11124 00000100:00080000:1.0:1787758069.953400:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758069.953405:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758069.953409:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758069.953416:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758069.953420:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758069.953425:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758069.953429:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758069.953432:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758069.953433:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758069.953439:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758069.953442:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758069.953446:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758069.953451:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758069.953456:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.953459:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758069.953462:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758069.953484:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758069.953486:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.953492:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758069.953496:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758069.953503:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.953523:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758069.953538:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659711104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758069.953554:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758069.953560:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.953585:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758069.953590:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758069.953595:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758069.953597:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758069.953599:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758069.953603:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758069.953606:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11124 00000100:00080000:1.0:1787758069.953610:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758069.953613:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758069.953615:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758069.953618:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758069.953620:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758069.953623:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758069.953627:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758069.953631:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758069.953633:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758069.953637:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758069.953641:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758069.953644:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758069.953654:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758069.953657:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.953659:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758069.953666:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758069.953668:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00000001:1.0:1787758069.953671:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.953678:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.953679:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787758069.953684:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758069.953686:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758069.953692:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.953695:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758069.953702:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758069.953703:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758069.953711:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758069.953719:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758069.953719:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.953731:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758069.953734:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.953743:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758069.953752:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659711232/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758069.953760:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659711104:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758069.953789:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758069.953797:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758069.953800:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.953817:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758069.953819:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758069.953824:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.953830:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.953832:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758069.953841:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758069.953847:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758069.953850:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758069.953853:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758069.953857:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758069.953859:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758069.953868:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00080000:1.0:1787758069.953875:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11124 00000100:00000001:0.0:1787758069.953879:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787758069.953886:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758069.953892:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758069.953895:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758069.953899:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758069.953903:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758069.953906:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758069.953911:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758069.953913:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758069.953915:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758069.953921:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758069.953926:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758069.953928:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758069.953934:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758069.953936:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758069.953949:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758069.953952:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758069.953962:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758069.953968:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.953978:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.953980:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787758069.953989:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758069.953994:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787758069.953995:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.954009:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758069.954012:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758069.954012:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758069.954020:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954027:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758069.954031:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758069.954046:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659711360/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758069.954048:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954070:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305797825 00000100:00000001:1.0:1787758069.954078:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758069.954082:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.954087:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954091:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758069.954092:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758069.954106:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758069.954108:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305797825 00000100:00000001:1.0:1787758069.954111:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758069.954116:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758069.954123:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758069.954129:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11124 00000100:00000001:0.0:1787758069.954130:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787758069.954138:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758069.954146:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758069.954147:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758069.954154:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758069.954171:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758069.954173:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758069.954175:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758069.954214:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758069.954221:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758069.954224:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758069.954227:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758069.954234:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758069.954238:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758069.954240:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758069.954244:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758069.954249:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:1.0:1787758069.954250:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.954260:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758069.954264:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758069.954270:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758069.954277:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758069/real 1787758069] req@ffff8ffb4a534a80 x1874588659711104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758069.954278:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758069.954292:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758069.954297:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.954305:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758069.954312:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758069.954320:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758069.954333:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758069.954339:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758069.954366:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:1.0:1787758069.954372:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659711488/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758069.954379:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758069.954390:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758069.954392:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758069.954399:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954402:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954409:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758069.954410:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954417:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758069.954434:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659711104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758069.954450:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954456:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758069.954462:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954464:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758069.954466:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954470:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758069.954473:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758069.954478:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758069.954484:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758069.954486:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758069.954491:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659711104:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758069.954498:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954501:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758069.954504:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758069.954507:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758069.954513:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758069.954516:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758069.954519:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758069.954524:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758069.954527:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758069.954530:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758069.954532:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954535:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954544:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758069.954547:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954550:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758069.954553:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758069.954555:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758069.954557:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758069.954559:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.954580:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758069.954585:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758069.954597:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758069.954604:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711232:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758069.954612:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758069.954616:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758069.954619:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.954623:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758069.954630:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954643:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758069.954647:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:0.0:1787758069.954655:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758069.954686:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758069.954694:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758069.954698:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758069.954700:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954702:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758069.954706:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954712:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305797134 00000100:00000001:0.0:1787758069.954716:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305797825 00000100:00000001:0.0:1787758069.954719:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305797134 00000100:00000001:0.0:1787758069.954723:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305797825 00000100:00000001:0.0:1787758069.954726:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758069.954729:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954731:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758069.954738:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758069/real 1787758069] req@ffff8ffb4a536300 x1874588659711232/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758069.954749:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758069.954751:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758069.954754:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758069.954756:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758069.954758:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954761:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758069.954762:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758069.954767:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659711232/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758069.954776:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954779:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758069.954782:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954784:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758069.954786:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954788:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758069.954805:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758069.954809:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758069.954813:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758069.954816:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758069.954820:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711232:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758069.954826:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758069.954828:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758069.954830:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.954834:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758069.954837:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758069.954839:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758069.954843:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711360:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758069.954847:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758069.954850:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758069.954852:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.954854:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758069.954858:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954864:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758069.954866:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758069.954871:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758069.954886:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758069.954891:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758069.954894:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758069.954896:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954898:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758069.954902:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954906:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305797131 00000100:00000001:0.0:1787758069.954909:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305797825 00000100:00000001:0.0:1787758069.954912:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305797131 00000100:00000001:0.0:1787758069.954914:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305797825 00000100:00000001:0.0:1787758069.954917:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758069.954919:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.954922:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758069.954926:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758069/real 1787758069] req@ffff8ffb4a534700 x1874588659711360/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758069.954936:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758069.954937:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758069.954940:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758069.954942:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758069.954944:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954946:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758069.954948:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758069.954954:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659711360/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758069.954967:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954971:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758069.954976:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954979:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758069.954982:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.954985:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758069.955005:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758069.955011:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758069.955019:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758069.955024:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758069.955031:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711360:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758069.955039:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758069.955043:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758069.955046:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.955050:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758069.955054:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758069.955058:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758069.955064:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711488:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758069.955068:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758069.955071:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758069.955074:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758069.955076:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758069.955080:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.955088:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758069.955090:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758069.955095:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758069.955114:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758069.955124:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758069.955128:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758069.955132:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.955135:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758069.955140:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.955147:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305797131 00000100:00000001:0.0:1787758069.955152:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305797825 00000100:00000001:0.0:1787758069.955156:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305797131 00000100:00000001:0.0:1787758069.955160:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305797825 00000100:00000001:0.0:1787758069.955164:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758069.955168:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.955172:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758069.955222:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758069/real 1787758069] req@ffff8ffb4a537800 x1874588659711488/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758069.955235:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758069.955237:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758069.955242:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758069.955244:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758069.955246:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955249:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758069.955251:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758069.955257:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659711488/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758069.955266:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955269:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758069.955272:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955274:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758069.955276:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955278:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758069.955296:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758069.955301:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758069.955307:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758069.955311:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758069.955317:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711488:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758069.955325:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955329:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758069.955331:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758069.955334:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758069.955339:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758069.955367:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758069.955370:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758069.955376:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758069.955379:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758069.955383:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758069.955386:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955390:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758069.955392:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758069.955395:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758069.955400:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758069.955402:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758069.955404:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758069.955409:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758069.955412:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758069.955414:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758069.955416:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955418:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758069.955420:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758069.955422:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758069.955424:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758069.955425:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758069.955427:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758069.955428:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758069.955430:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758069.955432:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758069.955433:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955436:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758069.955441:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758069.955443:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758069.955445:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758069.955447:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758070.529248:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758070.529258:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758070.529261:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758070.529267:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758070.529288:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758070.529293:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758070.529297:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758070.529301:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758070.529305:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758070.721309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758070.721320:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758070.721361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758070.721367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758070.721416:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758070.721420:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758070.721425:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758071.553265:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758071.553273:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758071.553275:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758071.553280:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758071.553285:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758071.553289:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758071.553291:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758071.553294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758071.553297:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758071.745435:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758071.745447:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758071.745457:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758071.745459:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758071.745556:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758071.745562:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758071.745567:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758072.577358:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758072.577367:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758072.577369:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758072.577375:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758072.577379:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758072.577383:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758072.577386:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758072.577389:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758072.577392:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758072.769404:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758072.769413:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758072.769421:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758072.769422:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758072.769527:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758072.769533:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758072.769538:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758073.601296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758073.601305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758073.601307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758073.601313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758073.601317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758073.601322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758073.601324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758073.601327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758073.601329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758073.793418:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758073.793426:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758073.793432:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758073.793436:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758073.793439:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758073.793443:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758073.793446:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758073.793449:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758073.793451:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758073.793454:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758073.793529:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758073.793535:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758073.793539:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758074.625387:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758074.625402:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758074.625405:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758074.625413:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758074.625418:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758074.625424:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758074.625427:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758074.625431:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758074.625435:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758074.817355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758074.817364:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758074.817371:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758074.817374:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758074.817409:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758074.817412:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758074.817415:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758075.073395:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758075.073409:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758075.073412:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758075.073417:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758075.073422:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758075.073432:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11129 00000100:00080000:2.0:1787758075.073445:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758075.073449:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758075.073452:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758075.073457:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758075.073460:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758075.073465:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758075.073468:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758075.073471:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758075.073472:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758075.073491:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758075.073495:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758075.073498:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758075.073502:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758075.073506:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073509:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758075.073512:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758075.073526:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758075.073529:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073533:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758075.073536:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758075.073541:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073556:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758075.073568:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659711616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758075.073582:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758075.073587:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073610:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758075.073614:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758075.073619:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758075.073621:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758075.073623:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758075.073627:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758075.073630:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11129 00000100:00080000:2.0:1787758075.073634:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758075.073636:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758075.073639:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758075.073642:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758075.073644:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758075.073646:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758075.073649:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758075.073652:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758075.073654:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758075.073657:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758075.073660:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758075.073663:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758075.073669:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758075.073672:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073675:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758075.073683:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758075.073687:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073690:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758075.073695:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758075.073706:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787758075.073707:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073713:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758075.073717:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758075.073718:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787758075.073720:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.073724:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758075.073731:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758075.073732:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.073734:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758075.073744:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659711744/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758075.073745:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758075.073753:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758075.073764:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073776:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758075.073782:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758075.073784:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659711616:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758075.073788:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758075.073798:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758075.073800:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787758075.073808:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787758075.073811:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787758075.073816:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787758075.073817:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758075.073826:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:2.0:1787758075.073827:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787758075.073833:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.073838:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758075.073850:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11129 00000100:00000001:0.0:1787758075.073857:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787758075.073859:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758075.073868:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758075.073871:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758075.073878:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000020:00000001:2.0:1787758075.073878:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758075.073885:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758075.073888:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787758075.073894:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758075.073903:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758075.073908:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758075.073912:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758075.073919:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758075.073926:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758075.073930:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758075.073937:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758075.073947:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758075.073948:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758075.073958:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758075.073965:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758075.073968:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00000001:2.0:1787758075.073972:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.073977:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758075.073981:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.073983:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.073990:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787758075.073995:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758075.073998:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758075.074000:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074008:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305802945 00000100:00000001:2.0:1787758075.074008:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758075.074013:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758075.074026:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:2.0:1787758075.074034:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074038:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305802945 00000100:00000001:2.0:1787758075.074048:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074050:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00100000:2.0:1787758075.074065:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659711872/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758075.074077:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758075.074080:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758075.074081:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.074086:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758075.074090:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758075.074103:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758075.074105:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758075.074106:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758075.074112:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758075.074116:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758075.074118:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758075.074128:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11129 00000100:00080000:2.0:1787758075.074135:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758075.074140:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758075/real 1787758075] req@ffff8ffb47a38380 x1874588659711616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758075.074141:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758075.074147:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758075.074156:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758075.074160:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758075.074164:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758075.074173:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758075.074178:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758075.074236:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758075.074246:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787758075.074249:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787758075.074271:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758075.074275:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758075.074280:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758075.074283:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758075.074289:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758075.074289:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758075.074297:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758075.074302:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074305:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758075.074306:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074316:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:2.0:1787758075.074317:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758075.074323:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074324:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758075.074327:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758075.074332:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758075.074333:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:2.0:1787758075.074336:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074338:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758075.074347:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758075.074355:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659711616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758075.074364:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659712000/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758075.074395:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758075.074402:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758075.074406:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758075.074408:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074416:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758075.074418:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074423:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758075.074429:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074433:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758075.074436:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758075.074440:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758075.074448:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758075.074453:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758075.074458:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659711616:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758075.074464:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074467:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758075.074470:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758075.074473:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758075.074478:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758075.074481:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758075.074484:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758075.074489:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758075.074491:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758075.074496:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758075.074498:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074501:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074509:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758075.074511:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074514:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758075.074516:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758075.074519:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758075.074520:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758075.074522:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758075.074526:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758075.074528:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758075.074531:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758075.074535:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711744:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758075.074540:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758075.074542:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758075.074545:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758075.074548:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758075.074552:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074561:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758075.074563:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:0.0:1787758075.074569:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758075.074588:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758075.074595:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758075.074598:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758075.074600:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074603:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758075.074606:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074612:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305802254 00000100:00000001:0.0:1787758075.074615:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305802945 00000100:00000001:0.0:1787758075.074618:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305802254 00000100:00000001:0.0:1787758075.074621:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305802945 00000100:00000001:0.0:1787758075.074624:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758075.074627:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074629:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758075.074635:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758075/real 1787758075] req@ffff8ffb47a38000 x1874588659711744/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758075.074646:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758075.074648:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758075.074650:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758075.074652:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758075.074654:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074656:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758075.074658:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758075.074663:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659711744/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758075.074672:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074675:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758075.074679:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074681:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758075.074682:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074685:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758075.074699:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758075.074703:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758075.074707:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758075.074710:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758075.074714:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711744:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758075.074718:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758075.074720:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758075.074722:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758075.074724:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758075.074726:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758075.074729:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758075.074732:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711872:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758075.074736:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758075.074738:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758075.074740:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758075.074742:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758075.074746:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074751:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758075.074753:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787758075.074757:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758075.074769:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758075.074774:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758075.074777:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758075.074779:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074780:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758075.074784:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074788:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305802251 00000100:00000001:0.0:1787758075.074791:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305802945 00000100:00000001:0.0:1787758075.074794:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305802251 00000100:00000001:0.0:1787758075.074797:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305802945 00000100:00000001:0.0:1787758075.074800:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758075.074802:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074805:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758075.074810:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758075/real 1787758075] req@ffff8ffb47a3b480 x1874588659711872/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758075.074818:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758075.074820:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758075.074822:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758075.074824:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758075.074826:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074828:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758075.074829:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758075.074833:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659711872/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758075.074842:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074845:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758075.074848:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074850:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758075.074851:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074853:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758075.074864:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758075.074867:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758075.074871:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758075.074874:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758075.074877:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659711872:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758075.074882:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758075.074884:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758075.074886:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758075.074889:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758075.074891:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758075.074894:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758075.074897:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712000:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758075.074900:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758075.074902:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758075.074904:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758075.074906:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758075.074909:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074914:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758075.074916:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787758075.074920:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758075.074930:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758075.074934:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758075.074937:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758075.074939:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074941:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758075.074944:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074947:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305802251 00000100:00000001:0.0:1787758075.074951:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305802945 00000100:00000001:0.0:1787758075.074954:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305802251 00000100:00000001:0.0:1787758075.074956:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305802945 00000100:00000001:0.0:1787758075.074959:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758075.074961:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.074964:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758075.074968:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758075/real 1787758075] req@ffff8ffb47a39180 x1874588659712000/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758075.074977:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758075.074978:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758075.074980:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758075.074982:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758075.074984:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.074987:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758075.074989:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758075.074994:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659712000/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758075.075003:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.075005:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758075.075008:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.075010:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758075.075011:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.075013:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758075.075025:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758075.075028:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758075.075031:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758075.075033:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758075.075037:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712000:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758075.075041:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.075043:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758075.075045:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758075.075046:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758075.075049:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758075.075050:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758075.075052:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758075.075054:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758075.075056:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758075.075058:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758075.075060:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.075062:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758075.075063:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758075.075065:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758075.075067:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758075.075069:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758075.075071:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758075.075073:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758075.075075:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758075.075080:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758075.075083:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.075086:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758075.075089:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758075.075091:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758075.075094:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758075.075096:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758075.075098:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758075.075101:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758075.075103:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758075.075107:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758075.075110:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.075114:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758075.075120:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758075.075123:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758075.075127:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758075.075130:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758075.649328:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758075.649346:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758075.649350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758075.649362:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758075.649370:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758075.649378:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758075.649383:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758075.649389:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758075.649395:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758075.841409:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758075.841418:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758075.841429:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758075.841431:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758075.841502:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758075.841510:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758075.841516:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758076.673327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758076.673337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758076.673339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758076.673345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758076.673349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758076.673354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758076.673356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758076.673360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758076.673362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758076.865414:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758076.865429:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758076.865435:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758076.865441:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758076.865448:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758076.865464:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758076.865468:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758076.865473:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758076.865481:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758076.865484:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758076.865566:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758076.865572:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758076.865577:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758077.697285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758077.697297:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758077.697300:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758077.697306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758077.697311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758077.697316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758077.697320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758077.697323:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758077.697327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758077.889307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758077.889313:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758077.889317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758077.889319:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758077.889350:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758077.889354:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758077.889356:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758078.721315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758078.721324:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758078.721326:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758078.721332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758078.721335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758078.721339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758078.721341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758078.721345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758078.721347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758078.913254:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758078.913260:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758078.913264:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758078.913266:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758078.913284:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758078.913286:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758078.913289:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758079.745323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758079.745336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758079.745339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758079.745348:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758079.745353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758079.745359:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758079.745362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758079.745366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758079.745370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758079.937356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758079.937371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758079.937382:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758079.937384:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758079.937455:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758079.937463:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758079.937469:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758080.193344:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758080.193357:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758080.193361:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758080.193365:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758080.193371:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758080.193381:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11134 00000100:00080000:3.0:1787758080.193394:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758080.193398:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758080.193401:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758080.193407:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758080.193410:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758080.193414:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758080.193418:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758080.193421:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758080.193423:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758080.193429:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758080.193432:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758080.193436:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758080.193440:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758080.193444:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.193447:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758080.193450:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758080.193464:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758080.193466:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.193470:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758080.193473:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758080.193478:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.193493:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758080.193505:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659712128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758080.193526:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758080.193534:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.193561:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758080.193568:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758080.193575:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758080.193578:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758080.193582:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758080.193588:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758080.193594:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11134 00000100:00080000:3.0:1787758080.193603:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758080.193610:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758080.193615:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758080.193620:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758080.193624:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758080.193628:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758080.193631:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758080.193635:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758080.193639:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758080.193641:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787758080.193649:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758080.193656:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758080.193661:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.193664:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758080.193671:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758080.193673:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758080.193676:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758080.193689:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758080.193690:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758080.193700:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758080.193707:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.193708:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.193726:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787758080.193736:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659712128:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1787758080.193741:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758080.193763:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:3.0:1787758080.193763:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.193778:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.193783:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758080.193788:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758080.193789:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758080.193798:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.193800:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758080.193808:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758080.193812:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758080.193817:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.193818:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.193855:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758080.193857:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.193860:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787758080.193871:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:3.0:1787758080.193885:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659712256/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758080.193907:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758080.193913:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.193923:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758080.193941:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758080.193944:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787758080.193958:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758080.193962:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758080.193966:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758080.193972:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758080.193977:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787758080.193979:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11134 00000100:00000001:0.0:1787758080.193987:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787758080.193990:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758080.193993:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.193998:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758080.194001:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758080.194005:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758080.194013:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758080.194015:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758080.194024:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758080.194032:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305808065 00000020:00000001:3.0:1787758080.194032:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758080.194045:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758080.194052:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758080.194055:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758080.194058:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787758080.194067:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758080.194074:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194075:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305808065 02000000:00000001:3.0:1787758080.194081:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758080.194089:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787758080.194091:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758080.194102:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194104:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758080.194107:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758080.194115:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787758080.194118:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758080.194125:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00000001:3.0:1787758080.194127:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.194133:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758080.194142:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758080/real 1787758080] req@ffff8ffb45c30000 x1874588659712128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758080.194144:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758080.194153:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.194226:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758080.194229:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758080.194235:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.194245:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758080.194274:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00100000:3.0:1787758080.194277:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659712384/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787758080.194281:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758080.194292:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758080.194304:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758080.194304:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758080.194312:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758080.194314:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194328:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758080.194330:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194334:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758080.194340:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758080.194353:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787758080.194355:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659712128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758080.194361:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758080.194372:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758080.194394:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758080.194406:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758080.194408:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11134 00000100:00000001:0.0:1787758080.194415:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758080.194418:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758080.194424:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758080.194427:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758080.194430:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758080.194434:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758080.194436:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.194450:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000020:00000001:3.0:1787758080.194453:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758080.194466:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758080.194469:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787758080.194472:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758080.194487:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758080.194492:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00080000:0.0:1787758080.194494:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787758080.194496:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758080.194508:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758080.194517:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758080.194518:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:3.0:1787758080.194527:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758080.194531:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1787758080.194539:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787758080.194551:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659712128:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787758080.194553:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.194565:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758080.194573:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758080.194576:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787758080.194586:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194587:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758080.194593:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:3.0:1787758080.194606:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.194607:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758080.194619:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:3.0:1787758080.194623:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000020:00000001:0.0:1787758080.194625:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:3.0:1787758080.194628:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758080.194632:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787758080.194635:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758080.194638:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758080.194642:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758080.194644:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.194648:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758080.194657:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758080.194659:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758080.194663:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194666:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.194685:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00100000:3.0:1787758080.194692:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659712512/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758080.194694:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194705:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758080.194712:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758080.194719:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758080.194722:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758080.194729:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.194739:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758080.194748:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758080.194749:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758080.194763:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.194764:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758080.194788:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758080.194790:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712256:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758080.194807:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758080.194811:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758080.194815:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.194820:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758080.194826:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194837:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758080.194839:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787758080.194846:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758080.194870:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758080.194878:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758080.194881:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758080.194883:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194886:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758080.194889:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194895:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305807374 00000100:00000001:0.0:1787758080.194900:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305808065 00000100:00000001:0.0:1787758080.194903:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305807374 00000100:00000001:0.0:1787758080.194906:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305808065 00000100:00000001:0.0:1787758080.194909:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758080.194911:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.194913:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758080.194919:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758080/real 1787758080] req@ffff8ffb45c31500 x1874588659712256/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758080.194933:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758080.194935:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758080.194938:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758080.194940:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758080.194942:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.194944:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758080.194946:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758080.194951:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659712256/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758080.194960:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.194963:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758080.194966:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.194968:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758080.194970:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.194972:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758080.194993:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758080.194997:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758080.195001:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758080.195004:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758080.195008:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712256:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758080.195012:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758080.195015:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758080.195017:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.195020:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758080.195022:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758080.195025:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758080.195029:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712384:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758080.195032:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758080.195034:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758080.195037:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.195040:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758080.195043:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195050:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758080.195051:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787758080.195057:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758080.195069:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758080.195074:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758080.195076:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758080.195078:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195080:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758080.195083:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195088:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305807371 00000100:00000001:0.0:1787758080.195091:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305808065 00000100:00000001:0.0:1787758080.195094:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305807371 00000100:00000001:0.0:1787758080.195097:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305808065 00000100:00000001:0.0:1787758080.195100:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758080.195102:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195104:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758080.195109:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758080/real 1787758080] req@ffff8ffb45c32a00 x1874588659712384/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758080.195118:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758080.195119:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758080.195122:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758080.195123:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758080.195125:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195127:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758080.195129:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758080.195133:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659712384/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758080.195141:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195144:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758080.195148:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195150:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758080.195152:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195154:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758080.195228:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758080.195233:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758080.195238:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758080.195241:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758080.195246:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712384:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758080.195251:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195254:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758080.195256:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758080.195258:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758080.195262:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758080.195264:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758080.195266:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758080.195269:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758080.195272:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758080.195275:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758080.195276:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195279:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758080.195280:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758080.195282:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758080.195285:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758080.195287:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758080.195288:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758080.195290:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758080.195291:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758080.195293:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758080.195295:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195297:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195301:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758080.195303:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195305:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758080.195307:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758080.195309:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758080.195311:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758080.195314:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.195317:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758080.195319:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758080.195322:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758080.195326:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712512:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758080.195330:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758080.195333:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758080.195335:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758080.195337:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758080.195341:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195348:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758080.195350:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787758080.195356:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758080.195372:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758080.195378:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758080.195381:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758080.195383:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195385:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758080.195389:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195394:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305807372 00000100:00000001:0.0:1787758080.195397:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305808065 00000100:00000001:0.0:1787758080.195401:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305807372 00000100:00000001:0.0:1787758080.195403:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305808065 00000100:00000001:0.0:1787758080.195406:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758080.195409:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195411:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758080.195416:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758080/real 1787758080] req@ffff8ffb45c30700 x1874588659712512/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758080.195425:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758080.195426:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758080.195429:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758080.195431:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758080.195433:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195437:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758080.195439:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758080.195443:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659712512/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758080.195453:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195455:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758080.195458:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195461:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758080.195463:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195466:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758080.195491:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758080.195496:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758080.195503:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758080.195507:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758080.195514:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712512:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758080.195522:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195525:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758080.195528:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758080.195532:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758080.195536:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758080.195539:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758080.195542:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758080.195546:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758080.195548:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758080.195552:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758080.195554:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195558:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758080.195564:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758080.195567:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758080.195571:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758080.195574:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758080.769309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758080.769322:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758080.769326:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758080.769335:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758080.769340:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758080.769346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758080.769350:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758080.769355:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758080.769359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758080.961414:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758080.961423:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758080.961431:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758080.961434:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758080.961516:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758080.961526:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758080.961531:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758081.793306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758081.793318:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758081.793321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758081.793329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758081.793334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758081.793340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758081.793343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758081.793347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758081.793351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758081.985359:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758081.985368:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758081.985375:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758081.985377:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758081.985451:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758081.985460:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758081.985468:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758082.817328:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758082.817345:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758082.817350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758082.817357:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758082.817360:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758082.817365:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758082.817368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758082.817371:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758082.817374:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758083.009333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758083.009342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758083.009350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758083.009351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758083.009378:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758083.009382:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758083.009386:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758083.841320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758083.841333:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758083.841336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758083.841343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758083.841349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758083.841355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758083.841359:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758083.841363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758083.841367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758084.033327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758084.033335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758084.033343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758084.033347:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758084.033351:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758084.033355:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758084.033359:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758084.033362:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758084.033364:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758084.033367:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758084.033437:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758084.033443:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758084.033448:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758084.865298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758084.865310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758084.865313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758084.865321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758084.865326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758084.865332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758084.865335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758084.865339:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758084.865343:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758085.057341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758085.057350:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758085.057358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758085.057360:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758085.057508:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758085.057514:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758085.057518:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758085.313323:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758085.313340:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758085.313345:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758085.313350:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758085.313358:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758085.313370:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11140 00000100:00080000:3.0:1787758085.313389:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758085.313393:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758085.313397:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758085.313403:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758085.313407:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758085.313412:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758085.313415:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758085.313418:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758085.313420:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758085.313427:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758085.313431:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.313434:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758085.313439:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758085.313444:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313447:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758085.313450:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.313469:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758085.313472:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313476:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758085.313481:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758085.313488:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313507:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758085.313523:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659712640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758085.313538:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758085.313544:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313575:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758085.313579:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758085.313585:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758085.313587:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758085.313589:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758085.313593:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758085.313596:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11140 00000100:00080000:3.0:1787758085.313600:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758085.313603:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758085.313604:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758085.313608:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758085.313610:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758085.313613:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758085.313615:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758085.313618:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758085.313620:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758085.313623:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758085.313626:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.313628:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758085.313636:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758085.313638:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313641:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758085.313647:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758085.313651:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313653:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.313658:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758085.313661:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313663:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758085.313665:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758085.313668:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313675:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758085.313684:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659712768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758085.313695:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758085.313698:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313701:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.313706:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787758085.313707:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758085.313732:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758085.313734:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758085.313735:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758085.313740:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758085.313741:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758085.313746:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758085.313748:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758085.313753:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758085.313754:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11140 00000100:00080000:3.0:1787758085.313762:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758085.313763:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758085.313765:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758085.313769:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758085.313770:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758085.313773:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758085.313796:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787758085.313798:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758085.313800:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758085.313806:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758085.313812:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:0.0:1787758085.313814:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659712640:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758085.313815:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758085.313829:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758085.313831:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758085.313834:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.313839:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758085.313841:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758085.313845:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758085.313849:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.313851:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758085.313859:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758085.313860:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758085.313868:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758085.313873:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758085.313874:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313877:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.313892:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758085.313894:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313915:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758085.313917:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758085.313921:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.313931:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758085.313933:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.313938:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00100000:3.0:1787758085.313947:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659712896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758085.313952:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758085.313985:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758085.313989:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.313994:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758085.313998:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758085.314004:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758085.314007:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758085.314009:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758085.314013:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758085.314016:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11140 00000100:00080000:3.0:1787758085.314036:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758085.314040:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758085.314041:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758085.314045:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758085.314046:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758085.314049:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758085.314052:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758085.314055:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758085.314057:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758085.314061:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758085.314064:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.314069:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758085.314096:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000800:00000100:0.0:1787758085.314106:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787758085.314106:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.314133:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758085.314136:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758085.314138:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.314140:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758085.314147:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758085.314149:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314150:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758085.314151:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758085.314156:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758085.314159:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787758085.314160:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314167:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314172:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758085.314172:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758085.314201:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659713024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758085.314203:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.314256:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758085.314260:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758085.314264:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314281:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305813186 00000100:00000001:0.0:1787758085.314304:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787758085.314313:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305813186 00000100:00000001:0.0:1787758085.314322:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758085.314329:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758085.314333:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314338:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758085.314349:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758085/real 1787758085] req@ffff8ffb45c30a80 x1874588659712640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758085.314366:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758085.314369:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758085.314374:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758085.314378:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758085.314381:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314385:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758085.314388:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758085.314394:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659712640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758085.314404:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314409:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758085.314414:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314417:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758085.314419:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314422:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758085.314426:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758085.314430:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758085.314435:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758085.314438:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758085.314443:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659712640:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758085.314449:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758085.314451:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758085.314453:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758085.314455:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758085.314458:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758085.314461:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758085.314465:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712768:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758085.314470:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758085.314473:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758085.314475:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758085.314477:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758085.314481:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314489:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758085.314491:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:0.0:1787758085.314498:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758085.314516:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758085.314522:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758085.314526:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758085.314528:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314530:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758085.314534:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314540:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305812494 00000100:00000001:0.0:1787758085.314544:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305813186 00000100:00000001:0.0:1787758085.314547:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305812494 00000100:00000001:0.0:1787758085.314552:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305813186 00000100:00000001:0.0:1787758085.314555:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758085.314558:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314560:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758085.314566:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758085/real 1787758085] req@ffff8ffb45c33b80 x1874588659712768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758085.314575:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758085.314577:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758085.314579:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758085.314582:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758085.314584:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314586:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758085.314588:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758085.314593:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659712768/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758085.314601:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314604:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758085.314606:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314608:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758085.314610:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314612:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758085.314633:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758085.314636:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758085.314641:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758085.314644:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758085.314648:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712768:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758085.314653:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314656:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758085.314658:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758085.314662:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758085.314669:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758085.314672:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758085.314675:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758085.314681:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758085.314684:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758085.314687:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758085.314689:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314691:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758085.314693:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758085.314695:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758085.314697:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758085.314698:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758085.314700:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758085.314702:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758085.314703:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758085.314705:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758085.314707:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314709:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314736:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758085.314738:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314741:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758085.314742:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758085.314743:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758085.314745:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758085.314746:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758085.314748:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758085.314750:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758085.314752:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758085.314756:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758085.314759:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758085.314761:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758085.314763:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758085.314764:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758085.314767:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314773:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758085.314774:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:0.0:1787758085.314778:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758085.314810:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758085.314815:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758085.314819:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758085.314821:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314823:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758085.314827:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314831:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305812491 00000100:00000001:0.0:1787758085.314834:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305813186 00000100:00000001:0.0:1787758085.314837:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305812491 00000100:00000001:0.0:1787758085.314840:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305813186 00000100:00000001:0.0:1787758085.314843:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758085.314846:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314848:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758085.314853:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758085/real 1787758085] req@ffff8ffb45c31c00 x1874588659712896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758085.314862:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758085.314864:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758085.314866:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758085.314868:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758085.314869:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314871:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758085.314873:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758085.314878:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659712896/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758085.314887:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314890:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758085.314909:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314911:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758085.314912:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.314914:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758085.314931:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758085.314934:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758085.314937:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758085.314940:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758085.314943:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659712896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758085.314947:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758085.314948:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758085.314950:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758085.314952:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758085.314953:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758085.314955:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758085.314958:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758085.314961:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758085.314963:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758085.314982:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758085.314984:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758085.314988:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.314994:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758085.314996:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:0.0:1787758085.315000:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758085.315013:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758085.315018:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758085.315021:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758085.315022:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.315025:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758085.315028:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.315032:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305812491 00000100:00000001:0.0:1787758085.315035:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305813186 00000100:00000001:0.0:1787758085.315038:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305812491 00000100:00000001:0.0:1787758085.315041:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305813186 00000100:00000001:0.0:1787758085.315044:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758085.315047:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.315049:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758085.315053:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758085/real 1787758085] req@ffff8ffb45c30e00 x1874588659713024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758085.315097:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758085.315099:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758085.315102:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758085.315105:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758085.315106:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.315109:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758085.315111:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758085.315133:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659713024/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758085.315141:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.315144:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758085.315146:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.315148:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758085.315149:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.315150:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758085.315168:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758085.315170:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758085.315173:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758085.315175:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758085.315178:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713024:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758085.315200:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.315203:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758085.315205:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758085.315207:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758085.315210:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758085.315212:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758085.315213:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758085.315243:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758085.315245:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758085.315248:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758085.315250:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.315253:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758085.315255:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758085.315257:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758085.315259:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758085.315261:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758085.315262:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758085.315264:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758085.315266:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758085.315267:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758085.315269:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.315271:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758085.315276:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758085.315277:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758085.315280:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758085.315282:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758085.889307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758085.889320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758085.889324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758085.889332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758085.889338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758085.889344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758085.889347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758085.889352:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758085.889356:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758086.081374:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758086.081388:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758086.081404:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758086.081407:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758086.081470:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758086.081477:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758086.081484:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758086.913316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758086.913328:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758086.913331:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758086.913340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758086.913345:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758086.913351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758086.913355:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758086.913360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758086.913364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758087.105283:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758087.105323:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758087.105327:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758087.105332:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758087.105341:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758087.105345:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758087.105350:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758087.105355:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758087.105363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758087.105367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758087.105408:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758087.105415:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758087.105421:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758087.937325:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758087.937337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758087.937341:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758087.937349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758087.937354:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758087.937360:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758087.937375:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758087.937379:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758087.937382:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758088.129355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758088.129367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758088.129377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758088.129379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758088.129469:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758088.129477:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758088.129483:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758088.961249:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758088.961255:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758088.961256:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758088.961260:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758088.961263:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758088.961266:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758088.961268:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758088.961271:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758088.961273:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758089.153261:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758089.153267:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758089.153271:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758089.153272:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758089.153313:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758089.153316:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758089.153319:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758089.985355:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758089.985363:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758089.985365:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758089.985371:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758089.985374:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758089.985378:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758089.985381:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758089.985384:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758089.985387:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758090.177290:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758090.177297:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758090.177302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758090.177304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758090.177408:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758090.177412:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758090.177416:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758090.433323:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758090.433336:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758090.433340:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758090.433344:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758090.433349:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758090.433358:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11145 00000100:00080000:1.0:1787758090.433371:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758090.433376:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758090.433378:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758090.433383:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758090.433386:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758090.433390:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758090.433393:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758090.433396:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758090.433397:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758090.433403:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758090.433406:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758090.433409:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758090.433412:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758090.433416:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433419:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758090.433421:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758090.433434:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758090.433436:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433440:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758090.433443:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758090.433448:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433463:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758090.433478:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659713152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758090.433498:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758090.433506:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433529:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758090.433535:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758090.433541:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758090.433544:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758090.433547:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758090.433553:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758090.433558:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11145 00000100:00080000:1.0:1787758090.433565:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758090.433569:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758090.433571:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758090.433575:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758090.433577:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758090.433581:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758090.433583:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758090.433586:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758090.433588:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758090.433593:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758090.433596:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758090.433597:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758090.433603:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758090.433606:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433611:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758090.433611:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758090.433619:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758090.433620:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:1.0:1787758090.433627:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433628:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758090.433635:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433636:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758090.433643:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758090.433647:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758090.433650:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758090.433652:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758090.433656:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758090.433658:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758090.433664:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758090.433666:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758090.433670:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758090.433679:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659713152:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758090.433682:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433691:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758090.433700:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:1.0:1787758090.433701:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659713280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758090.433707:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758090.433716:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758090.433725:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758090.433727:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433729:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433739:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758090.433743:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758090.433746:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758090.433749:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758090.433751:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758090.433755:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758090.433756:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758090.433760:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00080000:1.0:1787758090.433764:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11145 00000100:00000001:0.0:1787758090.433769:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:1.0:1787758090.433773:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758090.433778:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758090.433779:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758090.433782:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758090.433784:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758090.433786:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758090.433789:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758090.433792:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758090.433794:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758090.433798:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758090.433801:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758090.433803:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758090.433808:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758090.433810:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433812:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758090.433816:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758090.433822:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758090.433823:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758090.433826:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758090.433834:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758090.433837:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433839:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758090.433842:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787758090.433843:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758090.433850:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433854:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758090.433858:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433860:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433866:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:1.0:1787758090.433872:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659713408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758090.433876:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433887:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305818305 00000100:00000001:1.0:1787758090.433892:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758090.433897:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.433900:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.433903:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758090.433904:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758090.433915:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758090.433917:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305818305 00000100:00000001:1.0:1787758090.433918:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758090.433925:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758090.433929:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758090.433931:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758090.433939:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11145 00000100:00080000:1.0:1787758090.433944:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758090.433946:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758090.433948:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758090.433952:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787758090.433953:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758090.433960:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758090.433962:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758090.433968:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758090.433969:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:1.0:1787758090.433979:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758090.434004:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:0.0:1787758090.434005:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758090/real 1787758090] req@ffff8ffb4a537b80 x1874588659713152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758090.434008:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758090.434028:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787758090.434036:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787758090.434038:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758090.434043:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:1.0:1787758090.434047:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758090.434051:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:1.0:1787758090.434054:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758090.434057:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:1.0:1787758090.434061:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434063:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758090.434067:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758090.434070:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00100000:1.0:1787758090.434073:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758090.434075:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:1.0:1787758090.434082:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.434090:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758090.434091:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659713152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787758090.434097:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758090.434102:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.434110:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758090.434114:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758090.434115:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758090.434122:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434124:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758090.434130:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758090.434132:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434133:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758090.434139:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434144:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758090.434148:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1787758090.434150:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659713536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758090.434156:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758090.434172:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758090.434175:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758090.434178:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1787758090.434214:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758090.434221:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758090.434248:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659713152:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758090.434260:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434265:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758090.434268:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758090.434271:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758090.434279:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758090.434283:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758090.434286:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758090.434292:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758090.434295:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758090.434300:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758090.434301:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434305:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434313:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758090.434315:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434318:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758090.434320:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758090.434322:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758090.434324:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758090.434326:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758090.434329:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758090.434332:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758090.434334:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758090.434339:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713280:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758090.434343:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758090.434346:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758090.434348:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758090.434350:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758090.434354:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434362:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758090.434364:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:0.0:1787758090.434369:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758090.434388:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758090.434394:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758090.434397:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758090.434399:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434402:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758090.434407:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434414:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305817614 00000100:00000001:0.0:1787758090.434419:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305818305 00000100:00000001:0.0:1787758090.434423:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305817614 00000100:00000001:0.0:1787758090.434427:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305818305 00000100:00000001:0.0:1787758090.434431:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758090.434435:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434438:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758090.434446:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758090/real 1787758090] req@ffff8ffb4a535c00 x1874588659713280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758090.434465:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758090.434467:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758090.434472:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758090.434476:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758090.434478:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434482:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758090.434485:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758090.434492:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659713280/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758090.434506:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434510:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758090.434514:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434516:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758090.434518:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434520:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758090.434535:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758090.434539:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758090.434543:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758090.434546:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758090.434550:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713280:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758090.434556:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758090.434557:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758090.434559:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758090.434562:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758090.434565:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758090.434567:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758090.434570:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758090.434574:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758090.434576:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758090.434579:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758090.434581:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758090.434585:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434593:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758090.434595:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758090.434600:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758090.434616:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758090.434622:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758090.434624:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758090.434626:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434628:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758090.434631:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434635:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305817611 00000100:00000001:0.0:1787758090.434638:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305818305 00000100:00000001:0.0:1787758090.434641:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305817611 00000100:00000001:0.0:1787758090.434644:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305818305 00000100:00000001:0.0:1787758090.434647:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758090.434649:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434651:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758090.434656:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758090/real 1787758090] req@ffff8ffb4a537800 x1874588659713408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758090.434664:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758090.434666:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758090.434668:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758090.434670:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758090.434672:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434674:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758090.434676:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758090.434680:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659713408/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758090.434687:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434690:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758090.434693:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434695:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758090.434697:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434699:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758090.434713:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758090.434716:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758090.434719:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758090.434722:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758090.434725:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758090.434729:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758090.434731:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758090.434732:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758090.434734:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758090.434736:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758090.434738:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758090.434742:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758090.434746:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758090.434747:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758090.434749:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758090.434751:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758090.434754:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434759:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758090.434760:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758090.434764:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758090.434774:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758090.434778:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758090.434781:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758090.434783:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434784:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758090.434787:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434791:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305817611 00000100:00000001:0.0:1787758090.434794:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305818305 00000100:00000001:0.0:1787758090.434797:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305817611 00000100:00000001:0.0:1787758090.434799:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305818305 00000100:00000001:0.0:1787758090.434802:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758090.434804:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434806:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758090.434810:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758090/real 1787758090] req@ffff8ffb4a534700 x1874588659713536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758090.434818:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758090.434819:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758090.434821:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758090.434823:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758090.434824:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434826:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758090.434828:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758090.434831:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659713536/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758090.434840:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434842:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758090.434845:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434847:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758090.434848:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434850:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758090.434864:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758090.434866:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758090.434869:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758090.434871:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758090.434875:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713536:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758090.434879:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434882:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758090.434884:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758090.434886:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758090.434888:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758090.434890:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758090.434892:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758090.434895:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758090.434897:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758090.434900:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758090.434901:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434903:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758090.434905:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758090.434906:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758090.434908:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758090.434910:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758090.434911:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758090.434914:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758090.434915:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758090.434916:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758090.434918:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434919:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758090.434921:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758090.434922:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758090.434923:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758090.434924:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758090.434926:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758090.434927:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758090.434928:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758090.434930:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758090.434931:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434933:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758090.434937:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758090.434939:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758090.434941:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758090.434943:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758091.009333:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758091.009343:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758091.009346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758091.009352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758091.009356:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758091.009361:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758091.009364:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758091.009367:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758091.009370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758091.201351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758091.201360:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758091.201368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758091.201371:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758091.201413:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758091.201416:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758091.201419:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758092.033377:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758092.033390:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758092.033394:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758092.033405:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758092.033414:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758092.033421:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758092.033426:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758092.033433:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758092.033439:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758092.225299:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758092.225308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758092.225317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758092.225320:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758092.225422:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758092.225428:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758092.225432:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758093.057346:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758093.057378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758093.057383:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758093.057405:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758093.057412:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758093.057419:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758093.057424:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758093.057430:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758093.057435:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758093.249393:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758093.249403:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758093.249410:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758093.249427:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758093.249430:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758093.249434:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758093.249438:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758093.249440:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758093.249443:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758093.249446:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758093.249567:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758093.249573:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758093.249577:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758094.081293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758094.081302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758094.081305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758094.081328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758094.081334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758094.081341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758094.081347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758094.081364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758094.081368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758094.273333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758094.273340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758094.273347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758094.273349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758094.273447:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758094.273452:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758094.273457:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758095.105225:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758095.105236:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758095.105238:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758095.105244:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758095.105248:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758095.105253:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758095.105255:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758095.105258:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758095.105261:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758095.297311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758095.297326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758095.297340:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758095.297343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758095.297383:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758095.297389:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758095.297396:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758095.553550:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758095.553566:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758095.553573:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758095.553581:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758095.553590:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758095.553603:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11150 00000100:00080000:3.0:1787758095.553622:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758095.553628:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758095.553632:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758095.553640:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758095.553646:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758095.553651:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758095.553657:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758095.553662:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758095.553666:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758095.553677:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758095.553683:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.553689:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758095.553696:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758095.553703:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.553707:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758095.553713:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.553728:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758095.553735:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.553742:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758095.553747:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758095.553755:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.553774:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758095.553793:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659713664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758095.553816:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758095.553827:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.553861:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758095.553869:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758095.553882:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758095.553887:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758095.553923:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758095.553933:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758095.553943:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11150 00000100:00080000:3.0:1787758095.553957:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758095.553968:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758095.553969:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758095.553975:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758095.553986:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:3.0:1787758095.553989:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758095.553998:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:3.0:1787758095.553999:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758095.554009:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758095.554014:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.554026:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758095.554031:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758095.554037:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758095.554050:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659713664:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758095.554059:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758095.554067:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758095.554071:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.554075:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758095.554084:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.554111:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758095.554114:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:0.0:1787758095.554120:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758095.554248:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758095.554270:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758095.554277:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758095.554281:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.554284:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758095.554293:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.554301:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305823426 00000100:00000001:0.0:1787758095.554318:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787758095.554326:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305823426 00000100:00000001:0.0:1787758095.554346:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758095.554352:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758095.554361:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.554365:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758095.554379:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758095/real 1787758095] req@ffff8ffb45c30e00 x1874588659713664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758095.554397:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758095.554399:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758095.554405:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758095.554409:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758095.554412:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.554416:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758095.554419:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758095.554426:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659713664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758095.554437:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.554442:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758095.554446:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.554448:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758095.554450:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.554455:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758095.554459:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758095.554464:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758095.554471:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758095.554475:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758095.554481:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659713664:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758095.554489:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.554495:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:3.0:1787758095.554496:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758095.554503:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000020:00000001:3.0:1787758095.554514:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:0.0:1787758095.554515:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758095.554527:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787758095.554528:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000020:00000001:0.0:1787758095.554533:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758095.554540:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758095.554549:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758095.554553:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758095.554561:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758095.554564:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.554570:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.554586:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758095.554591:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.554597:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758095.554601:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.554628:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758095.554649:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758095.554658:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.554663:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758095.554674:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758095.554682:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.554687:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758095.554695:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758095.554703:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.554710:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.554723:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758095.554727:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.554731:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758095.554734:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758095.554738:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.554749:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758095.554763:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659713792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758095.554795:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758095.554800:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.554825:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758095.554832:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758095.554839:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758095.554842:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758095.554848:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758095.554856:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758095.554863:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11150 00000100:00000001:0.0:1787758095.554866:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758095.554871:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758095.554875:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758095.554878:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787758095.554878:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758095.554886:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.554916:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758095.554917:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758095.554922:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758095.554925:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758095.554935:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758095.554938:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758095.554942:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758095.554949:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00100000:0.0:1787758095.554956:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713792:192.168.201.118@tcp:38:kworker.0 00000020:00000001:3.0:1787758095.554959:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758095.554973:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758095.554978:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758095.554981:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758095.554988:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758095.554996:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758095.554999:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787758095.555002:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758095.555012:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.555014:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.555024:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758095.555030:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787758095.555034:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758095.555036:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 02000000:00000001:3.0:1787758095.555050:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555052:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758095.555061:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758095.555068:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758095.555075:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758095.555083:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758095.555083:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555097:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787758095.555102:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758095.555104:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758095.555111:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.555112:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555116:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758095.555125:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758095.555126:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.555137:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758095.555138:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305822734 00000100:00000001:0.0:1787758095.555151:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305823426 02000000:00000001:3.0:1787758095.555153:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555156:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305822734 00000100:00000001:0.0:1787758095.555170:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305823426 00000100:00000001:0.0:1787758095.555174:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758095.555177:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.555180:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555181:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758095.555230:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758095/real 1787758095] req@ffff8ffb45c31c00 x1874588659713792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758095.555255:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00100000:3.0:1787758095.555256:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659713920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787758095.555259:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758095.555266:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758095.555272:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758095.555278:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.555292:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758095.555300:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758095.555301:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758095.555312:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758095.555318:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659713792/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758095.555330:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555333:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758095.555342:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758095.555344:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758095.555354:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.555361:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787758095.555364:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758095.555366:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.555374:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758095.555376:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758095.555384:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758095.555391:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758095.555403:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11150 00000100:00000001:0.0:1787758095.555406:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758095.555416:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758095.555418:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758095.555427:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758095.555432:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758095.555434:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787758095.555441:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787758095.555444:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713792:192.168.201.118@tcp:38:kworker.0 00000020:00000001:3.0:1787758095.555459:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758095.555462:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787758095.555468:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758095.555469:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758095.555478:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000020:00000001:3.0:1787758095.555482:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:0.0:1787758095.555483:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758095.555489:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758095.555498:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758095.555498:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:0.0:1787758095.555506:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787758095.555511:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758095.555514:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758095.555518:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758095.555518:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758095.555527:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758095.555534:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758095.555543:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758095.555546:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758095.555554:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555564:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787758095.555565:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758095.555578:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.555582:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758095.555589:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787758095.555592:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555598:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758095.555600:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758095.555606:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00100000:3.0:1787758095.555609:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758095.555615:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:3.0:1787758095.555620:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555623:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.555628:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.555634:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758095.555642:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758095.555643:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758095.555653:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758095.555654:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758095.555667:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:0.0:1787758095.555670:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713920:192.168.201.118@tcp:8:kworker.0 02000000:00000001:3.0:1787758095.555674:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758095.555685:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758095.555686:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.555691:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758095.555696:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.555699:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758095.555701:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.555706:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555719:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:3.0:1787758095.555725:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659714048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758095.555726:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:0.0:1787758095.555742:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758095.555759:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758095.555766:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758095.555769:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758095.555791:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758095.555793:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555797:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758095.555801:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555807:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758095.555812:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555818:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305822732 00000100:00000001:0.0:1787758095.555823:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305823426 00000100:00000001:0.0:1787758095.555826:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305822732 00000100:00000001:0.0:1787758095.555830:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305823426 00000100:00000001:0.0:1787758095.555833:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758095.555838:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.555842:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758095.555849:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758095/real 1787758095] req@ffff8ffb45c33b80 x1874588659713920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758095.555861:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758095.555864:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758095.555869:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758095.555872:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758095.555874:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.555878:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758095.555881:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758095.555888:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659713920/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758095.555924:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.555930:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758095.555937:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.555941:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758095.555944:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.555949:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758095.556012:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758095.556017:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758095.556026:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758095.556031:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758095.556040:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659713920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758095.556049:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556052:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758095.556054:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758095.556057:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758095.556061:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758095.556064:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758095.556066:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758095.556071:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758095.556072:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758095.556077:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758095.556082:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556086:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556093:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758095.556096:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.556100:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758095.556102:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758095.556104:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758095.556107:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758095.556109:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.556114:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758095.556117:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758095.556121:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758095.556127:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758095.556133:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758095.556137:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758095.556141:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758095.556145:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758095.556149:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.556159:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758095.556163:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787758095.556170:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758095.556246:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758095.556254:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758095.556258:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758095.556260:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.556262:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758095.556266:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.556271:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305822733 00000100:00000001:0.0:1787758095.556274:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305823426 00000100:00000001:0.0:1787758095.556278:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305822733 00000100:00000001:0.0:1787758095.556281:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305823426 00000100:00000001:0.0:1787758095.556284:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758095.556286:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.556289:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758095.556294:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758095/real 1787758095] req@ffff8ffb45c30a80 x1874588659714048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758095.556304:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758095.556305:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758095.556308:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758095.556310:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758095.556312:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556315:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758095.556317:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758095.556322:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659714048/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758095.556331:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556333:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758095.556336:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556338:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758095.556340:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556342:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758095.556362:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758095.556366:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758095.556370:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758095.556373:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758095.556377:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714048:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758095.556381:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556383:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758095.556385:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758095.556386:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758095.556389:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758095.556391:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758095.556392:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758095.556395:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758095.556397:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758095.556399:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758095.556401:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556403:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758095.556407:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758095.556409:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758095.556412:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758095.556414:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758096.129301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758096.129313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758096.129315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758096.129323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758096.129328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758096.129333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758096.129337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758096.129341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758096.129345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758096.321425:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758096.321444:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758096.321449:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758096.321455:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758096.321464:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758096.321468:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758096.321471:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758096.321474:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758096.321480:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758096.321482:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758096.321574:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758096.321583:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758096.321589:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758097.153281:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758097.153291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758097.153294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758097.153301:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758097.153305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758097.153310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758097.153313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758097.153317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758097.153320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758097.345325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758097.345332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758097.345339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758097.345341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758097.345478:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758097.345484:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758097.345489:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758098.177309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758098.177317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758098.177337:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758098.177343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758098.177347:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758098.177351:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758098.177354:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758098.177358:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758098.177361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758098.369368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758098.369392:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758098.369398:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758098.369399:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758098.369512:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758098.369515:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758098.369519:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758099.201291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758099.201301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758099.201304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758099.201311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758099.201316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758099.201320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758099.201324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758099.201327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758099.201331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758099.393321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758099.393330:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758099.393338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758099.393341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758099.393369:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758099.393373:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758099.393376:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758100.225250:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758100.225257:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758100.225260:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758100.225266:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758100.225270:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758100.225274:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758100.225276:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758100.225279:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758100.225282:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758100.417259:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758100.417264:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758100.417267:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758100.417268:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758100.417389:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758100.417397:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758100.417403:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758100.673425:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758100.673449:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758100.673453:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758100.673458:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758100.673465:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758100.673476:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11155 00000100:00080000:3.0:1787758100.673493:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758100.673497:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758100.673500:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758100.673506:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758100.673510:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758100.673516:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758100.673520:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758100.673523:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758100.673526:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758100.673533:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758100.673537:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758100.673541:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758100.673546:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758100.673551:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673555:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758100.673559:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758100.673579:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758100.673581:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673586:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758100.673591:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758100.673598:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673618:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758100.673633:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659714176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758100.673648:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758100.673654:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673674:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758100.673679:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758100.673683:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758100.673685:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758100.673687:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758100.673691:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758100.673694:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11155 00000100:00080000:3.0:1787758100.673698:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758100.673701:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758100.673703:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758100.673706:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758100.673708:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758100.673711:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758100.673714:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758100.673716:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758100.673718:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758100.673722:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758100.673724:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758100.673727:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758100.673734:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758100.673737:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673740:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758100.673745:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758100.673750:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673753:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758100.673757:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758100.673759:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673762:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758100.673764:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758100.673766:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.673775:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758100.673782:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.673783:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758100.673791:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758100.673797:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00100000:3.0:1787758100.673797:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659714304/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758100.673836:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758100.673849:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758100.673853:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758100.673856:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758100.673858:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673865:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758100.673871:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659714176:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758100.673878:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.673879:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758100.673885:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787758100.673889:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758100.673890:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758100.673903:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787758100.673905:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758100.673910:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758100.673914:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787758100.673915:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.673931:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758100.673939:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11155 00000100:00000001:0.0:1787758100.673944:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787758100.673945:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758100.673948:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:3.0:1787758100.673951:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758100.673957:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758100.673957:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758100.673971:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758100.673974:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758100.673977:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758100.673981:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758100.673984:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758100.673986:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758100.673992:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758100.673996:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758100.673998:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758100.674003:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000800:00000100:0.0:1787758100.674014:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787758100.674015:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.674023:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758100.674028:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758100.674032:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674033:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758100.674052:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758100.674059:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758100.674061:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787758100.674063:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674066:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.674068:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758100.674084:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787758100.674084:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758100.674104:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674111:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.674117:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674121:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305828545 00000100:00100000:3.0:1787758100.674133:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659714432/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758100.674141:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758100.674163:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758100.674164:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305828545 00000100:00000001:3.0:1787758100.674171:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674194:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758100.674203:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758100.674206:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674209:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758100.674228:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758100.674232:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758100.674237:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758100.674239:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758100.674242:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758100.674247:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758100.674252:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11155 00000100:00080000:3.0:1787758100.674260:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758100.674263:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758100.674270:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787758100.674272:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758100/real 1787758100] req@ffff8ffb45c30700 x1874588659714176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787758100.674274:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758100.674279:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758100.674284:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758100.674292:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758100.674306:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758100.674310:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758100.674325:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000400:00000001:0.0:1787758100.674328:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787758100.674331:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758100.674334:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787758100.674336:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758100.674343:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787758100.674344:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758100.674353:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674355:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758100.674357:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758100.674360:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:3.0:1787758100.674362:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758100.674367:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00000001:3.0:1787758100.674368:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674374:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758100.674374:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758100.674397:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:0.0:1787758100.674399:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659714176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758100.674401:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.674411:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758100.674416:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758100.674420:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.674431:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674439:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758100.674444:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659714560/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758100.674449:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758100.674466:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758100.674468:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758100.674477:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758100.674477:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758100.674486:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674487:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674495:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758100.674498:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758100.674505:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758100.674515:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758100.674519:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758100.674524:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659714176:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758100.674530:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674532:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758100.674534:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758100.674536:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758100.674541:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758100.674544:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758100.674547:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758100.674550:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758100.674552:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758100.674555:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758100.674556:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674559:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674566:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758100.674568:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674570:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758100.674572:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758100.674574:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758100.674576:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758100.674577:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758100.674580:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758100.674582:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758100.674584:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758100.674587:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714304:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758100.674590:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758100.674592:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758100.674594:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758100.674596:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758100.674599:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674606:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758100.674607:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787758100.674611:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758100.674626:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758100.674631:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758100.674633:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758100.674634:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674635:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758100.674638:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674642:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305827854 00000100:00000001:0.0:1787758100.674645:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305828545 00000100:00000001:0.0:1787758100.674648:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305827854 00000100:00000001:0.0:1787758100.674650:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305828545 00000100:00000001:0.0:1787758100.674652:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758100.674654:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674656:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758100.674661:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758100/real 1787758100] req@ffff8ffb45c32a00 x1874588659714304/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758100.674670:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758100.674672:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758100.674675:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758100.674677:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758100.674678:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674680:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758100.674682:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758100.674686:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659714304/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758100.674692:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674693:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758100.674695:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674696:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758100.674697:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674698:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758100.674710:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758100.674711:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758100.674714:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758100.674715:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758100.674718:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714304:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758100.674721:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758100.674722:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758100.674724:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758100.674726:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758100.674727:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758100.674728:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758100.674730:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758100.674732:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758100.674734:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758100.674735:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758100.674736:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758100.674738:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674743:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758100.674744:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787758100.674747:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758100.674755:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758100.674759:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758100.674760:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758100.674779:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674781:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758100.674785:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674789:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305827851 00000100:00000001:0.0:1787758100.674793:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305828545 00000100:00000001:0.0:1787758100.674796:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305827851 00000100:00000001:0.0:1787758100.674799:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305828545 00000100:00000001:0.0:1787758100.674845:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758100.674847:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674848:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758100.674851:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758100/real 1787758100] req@ffff8ffb45c31500 x1874588659714432/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758100.674857:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758100.674858:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758100.674860:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758100.674861:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758100.674862:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674864:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758100.674865:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758100.674867:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659714432/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758100.674872:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674873:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758100.674875:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674876:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758100.674876:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674878:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758100.674887:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758100.674889:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758100.674891:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758100.674892:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758100.674894:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758100.674897:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758100.674898:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758100.674899:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758100.674900:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758100.674902:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758100.674903:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758100.674905:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758100.674907:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758100.674909:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758100.674910:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758100.674911:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758100.674913:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674917:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758100.674917:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787758100.674920:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758100.674927:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758100.674930:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758100.674932:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758100.674933:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674934:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758100.674936:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674938:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305827851 00000100:00000001:0.0:1787758100.674940:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305828545 00000100:00000001:0.0:1787758100.674941:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305827851 00000100:00000001:0.0:1787758100.674943:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305828545 00000100:00000001:0.0:1787758100.674945:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758100.674946:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.674947:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758100.674949:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758100/real 1787758100] req@ffff8ffb45c30000 x1874588659714560/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758100.674954:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758100.674954:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758100.674956:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758100.674957:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758100.674957:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674958:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758100.674959:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758100.674961:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659714560/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758100.674966:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674967:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758100.674969:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674970:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758100.674971:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674972:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758100.674981:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758100.674982:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758100.674985:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758100.674986:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758100.674988:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714560:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758100.674991:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.674992:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758100.674993:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758100.674994:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758100.674995:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758100.674996:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758100.674997:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758100.674998:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758100.674999:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758100.675000:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758100.675001:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.675002:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758100.675003:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758100.675004:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758100.675005:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758100.675006:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758100.675007:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758100.675008:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758100.675008:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758100.675009:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758100.675010:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.675011:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758100.675011:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758100.675012:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758100.675013:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758100.675014:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758100.675014:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758100.675015:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758100.675016:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758100.675017:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758100.675017:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.675019:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758100.675021:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758100.675022:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758100.675023:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758100.675024:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758101.249298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758101.249307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758101.249310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758101.249316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758101.249321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758101.249325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758101.249329:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758101.249332:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758101.249336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758101.441350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758101.441359:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758101.441370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758101.441373:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758101.441466:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758101.441472:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758101.441477:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758102.273357:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758102.273366:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758102.273369:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758102.273376:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758102.273381:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758102.273386:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758102.273389:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758102.273392:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758102.273395:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758102.465370:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758102.465380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758102.465388:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758102.465389:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758102.465465:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758102.465471:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758102.465476:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758103.297286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758103.297299:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758103.297302:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758103.297310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758103.297315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758103.297320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758103.297324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758103.297329:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758103.297333:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758103.489385:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758103.489400:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758103.489413:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758103.489420:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758103.489425:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758103.489432:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758103.489439:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758103.489443:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758103.489448:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758103.489452:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758103.489490:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758103.489494:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758103.489498:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758104.321361:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758104.321373:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758104.321376:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758104.321384:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758104.321389:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758104.321395:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758104.321399:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758104.321403:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758104.321408:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758104.513314:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758104.513323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758104.513333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758104.513335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758104.513433:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758104.513437:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758104.513441:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758105.345358:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758105.345369:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758105.345373:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758105.345380:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758105.345386:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758105.345391:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758105.345395:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758105.345400:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758105.345403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758105.537315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758105.537325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758105.537334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758105.537337:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758105.537478:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758105.537483:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758105.537487:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758105.793421:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758105.793437:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758105.793441:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758105.793447:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758105.793454:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758105.793465:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11160 00000100:00080000:1.0:1787758105.793483:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758105.793488:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758105.793491:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758105.793497:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758105.793502:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758105.793507:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758105.793511:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758105.793514:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758105.793516:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758105.793522:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758105.793525:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758105.793529:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758105.793535:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758105.793542:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.793547:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758105.793551:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758105.793575:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758105.793583:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.793589:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758105.793595:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758105.793604:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.793628:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758105.793647:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659714688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758105.793670:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758105.793678:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.793709:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758105.793739:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758105.793746:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758105.793750:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758105.793753:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758105.793759:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758105.793765:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11160 00000100:00080000:1.0:1787758105.793773:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758105.793774:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758105.793783:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758105.793785:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758105.793788:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758105.793791:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:1.0:1787758105.793801:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758105.793802:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758105.793809:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.793818:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758105.793822:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758105.793826:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758105.793837:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659714688:192.168.201.118@tcp:250:kworker.0 00000020:00000001:1.0:1787758105.793845:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758105.793846:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:1.0:1787758105.793856:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:0.0:1787758105.793859:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:1.0:1787758105.793872:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:0.0:1787758105.793874:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.793893:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758105.793895:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758105.793905:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758105.793924:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758105.793927:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.793929:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758105.793931:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758105.793946:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758105.793952:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.793955:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758105.793958:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00100000:1.0:1787758105.793961:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758105.793966:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000008:00000001:1.0:1787758105.793970:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.793977:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.793984:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787758105.793988:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758105.793993:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.793996:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758105.793998:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758105.794001:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.794021:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758105.794029:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659714816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758105.794041:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758105.794044:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.794050:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758105.794055:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758105.794061:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758105.794063:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758105.794065:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758105.794068:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758105.794072:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11160 00000100:00080000:1.0:1787758105.794078:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758105.794081:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758105.794082:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758105.794089:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758105.794104:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000800:00000100:0.0:1787758105.794114:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:1.0:1787758105.794114:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758105.794129:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758105.794133:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758105.794135:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758105.794141:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758105.794143:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758105.794147:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758105.794152:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758105.794153:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787758105.794158:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758105.794161:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758105.794164:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794167:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758105.794174:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758105.794182:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794208:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305833666 00000008:00100000:1.0:1787758105.794256:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758105.794263:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.794267:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794269:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787758105.794280:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305833666 02000000:00000001:1.0:1787758105.794280:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758105.794285:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.794291:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758105.794293:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787758105.794303:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758105.794307:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758105.794309:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794313:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794319:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758105.794320:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758105.794334:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659714944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758105.794336:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758105/real 1787758105] req@ffff8ffb4a536300 x1874588659714688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758105.794374:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758105.794380:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.794387:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758105.794397:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758105.794406:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000400:00000001:0.0:1787758105.794409:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787758105.794411:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758105.794420:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000400:00000001:0.0:1787758105.794421:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758105.794425:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758105.794435:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:1.0:1787758105.794436:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11160 00000100:00000001:0.0:1787758105.794449:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:1.0:1787758105.794452:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758105.794453:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758105.794466:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758105.794468:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758105.794470:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758105.794473:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787758105.794479:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758105.794484:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758105.794487:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758105.794496:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:0.0:1787758105.794499:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659714688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758105.794501:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758105.794507:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758105.794521:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758105.794529:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758105.794544:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758105.794553:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758105.794556:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758105.794563:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794566:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758105.794574:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758105.794575:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.794584:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758105.794587:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:1.0:1787758105.794587:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758105.794600:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758105.794609:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00000001:1.0:1787758105.794611:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758105.794625:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787758105.794627:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758105.794634:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758105.794639:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758105.794644:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659714688:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787758105.794644:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758105.794650:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794659:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758105.794659:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758105.794667:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758105.794669:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787758105.794676:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794678:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758105.794687:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:1.0:1787758105.794693:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.794696:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758105.794702:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758105.794710:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00100000:1.0:1787758105.794737:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659715072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758105.794741:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758105.794749:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758105.794757:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758105.794763:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758105.794770:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758105.794771:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758105.794779:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758105.794786:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794789:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758105.794793:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794797:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758105.794802:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758105.794805:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758105.794809:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758105.794811:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.794816:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758105.794818:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758105.794822:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758105.794828:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714816:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758105.794834:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758105.794838:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758105.794842:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.794843:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758105.794848:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794856:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758105.794857:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:0.0:1787758105.794864:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758105.794883:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758105.794889:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758105.794891:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758105.794893:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794895:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758105.794898:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794919:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305832974 00000100:00000001:0.0:1787758105.794923:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305833666 00000100:00000001:0.0:1787758105.794926:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305832974 00000100:00000001:0.0:1787758105.794929:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305833666 00000100:00000001:0.0:1787758105.794932:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758105.794935:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.794937:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758105.794944:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758105/real 1787758105] req@ffff8ffb4a534a80 x1874588659714816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758105.794957:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758105.794960:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758105.794964:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758105.794967:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758105.794970:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.794974:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758105.794976:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758105.794985:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659714816/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758105.795000:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795005:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758105.795026:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795029:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758105.795031:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795033:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758105.795053:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758105.795058:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758105.795064:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758105.795068:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758105.795071:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714816:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758105.795076:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758105.795078:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758105.795080:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.795083:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758105.795085:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758105.795087:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758105.795090:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758105.795094:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758105.795097:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758105.795099:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.795100:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758105.795104:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795111:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758105.795112:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758105.795116:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758105.795131:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758105.795136:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758105.795139:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758105.795141:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795142:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758105.795145:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795149:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305832971 00000100:00000001:0.0:1787758105.795152:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305833666 00000100:00000001:0.0:1787758105.795155:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305832971 00000100:00000001:0.0:1787758105.795157:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305833666 00000100:00000001:0.0:1787758105.795160:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758105.795162:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795164:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758105.795168:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758105/real 1787758105] req@ffff8ffb4a534700 x1874588659714944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758105.795176:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758105.795177:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758105.795179:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758105.795181:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758105.795182:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795184:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758105.795186:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758105.795190:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659714944/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758105.795196:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795199:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758105.795201:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795203:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758105.795204:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795206:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758105.795244:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758105.795248:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758105.795251:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758105.795254:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758105.795258:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659714944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758105.795261:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758105.795263:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758105.795265:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.795268:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758105.795269:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758105.795271:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758105.795274:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758105.795278:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758105.795279:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758105.795281:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758105.795283:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758105.795286:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795291:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758105.795293:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758105.795297:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758105.795308:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758105.795313:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758105.795315:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758105.795317:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795318:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758105.795321:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795325:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305832972 00000100:00000001:0.0:1787758105.795328:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305833666 00000100:00000001:0.0:1787758105.795330:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305832972 00000100:00000001:0.0:1787758105.795332:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305833666 00000100:00000001:0.0:1787758105.795335:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758105.795336:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795338:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758105.795343:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758105/real 1787758105] req@ffff8ffb4a537800 x1874588659715072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758105.795355:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758105.795356:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758105.795360:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758105.795362:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758105.795364:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795368:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758105.795370:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758105.795376:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659715072/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758105.795388:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795392:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758105.795396:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795399:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758105.795401:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795404:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758105.795422:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758105.795427:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758105.795431:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758105.795434:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758105.795438:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715072:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758105.795442:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795444:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758105.795446:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758105.795447:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758105.795450:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758105.795452:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758105.795454:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758105.795456:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758105.795458:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758105.795460:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758105.795461:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795463:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758105.795464:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758105.795466:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758105.795467:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758105.795468:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758105.795469:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758105.795471:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758105.795472:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758105.795473:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758105.795476:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795478:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758105.795479:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758105.795481:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758105.795484:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758105.795486:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758105.795488:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758105.795491:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758105.795493:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758105.795496:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758105.795498:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795501:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758105.795507:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758105.795509:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758105.795512:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758105.795515:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758106.369378:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758106.369389:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758106.369393:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758106.369401:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758106.369407:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758106.369412:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758106.369416:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758106.369420:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758106.369424:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758106.561439:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758106.561451:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758106.561454:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758106.561459:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758106.561465:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758106.561468:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758106.561470:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758106.561472:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758106.561477:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758106.561478:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758106.561558:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758106.561564:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758106.561568:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758107.393383:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758107.393392:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758107.393394:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758107.393401:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758107.393405:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758107.393409:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758107.393412:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758107.393416:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758107.393420:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758107.585341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758107.585349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758107.585356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758107.585375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758107.585395:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758107.585397:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758107.585399:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758108.417316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758108.417325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758108.417328:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758108.417335:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758108.417339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758108.417343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758108.417346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758108.417350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758108.417353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758108.609387:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758108.609395:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758108.609402:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758108.609404:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758108.609497:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758108.609505:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758108.609513:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758109.441261:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758109.441268:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758109.441271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758109.441276:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758109.441279:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758109.441283:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758109.441286:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758109.441289:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758109.441291:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758109.633323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758109.633332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758109.633341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758109.633344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758109.633447:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758109.633454:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758109.633459:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758110.465339:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758110.465350:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758110.465352:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758110.465359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758110.465364:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758110.465368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758110.465371:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758110.465375:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758110.465378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758110.657336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758110.657348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758110.657358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758110.657361:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758110.657439:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758110.657445:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758110.657452:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758110.913291:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758110.913300:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758110.913302:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758110.913305:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758110.913309:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758110.913316:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11165 00000100:00080000:1.0:1787758110.913341:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758110.913344:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758110.913345:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758110.913348:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758110.913351:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758110.913353:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758110.913355:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758110.913356:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758110.913357:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758110.913361:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758110.913363:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758110.913364:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758110.913367:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758110.913369:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913371:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758110.913373:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758110.913397:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758110.913399:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913402:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758110.913405:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758110.913409:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913419:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758110.913427:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659715200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758110.913435:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758110.913439:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913455:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758110.913458:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758110.913461:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758110.913462:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758110.913463:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758110.913466:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758110.913467:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11165 00000100:00080000:1.0:1787758110.913470:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758110.913472:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758110.913473:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758110.913474:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758110.913475:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758110.913477:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758110.913479:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758110.913480:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758110.913481:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758110.913483:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758110.913484:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758110.913485:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758110.913489:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758110.913495:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913497:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758110.913501:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758110.913504:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913506:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758110.913509:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758110.913510:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913511:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758110.913512:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758110.913513:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913517:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758110.913521:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659715328/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758110.913528:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758110.913529:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913532:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758110.913533:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758110.913536:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787758110.913537:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758110.913541:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758110.913543:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:1.0:1787758110.913543:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758110.913548:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758110.913550:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758110.913551:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758110.913555:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758110.913558:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11165 02000000:00000001:0.0:1787758110.913560:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758110.913565:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758110.913567:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758110.913569:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913588:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758110.913592:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787758110.913595:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659715200:192.168.201.118@tcp:250:kworker.0 00000020:00000001:1.0:1787758110.913596:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758110.913599:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758110.913604:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758110.913606:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:1.0:1787758110.913609:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787758110.913612:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758110.913614:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758110.913616:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913618:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758110.913639:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758110.913642:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758110.913644:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758110.913647:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758110.913650:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913652:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787758110.913654:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00100000:1.0:1787758110.913654:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758110.913658:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913659:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758110.913662:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758110.913664:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758110.913668:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913669:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758110.913671:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758110.913672:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913678:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758110.913682:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659715456/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758110.913683:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758110.913688:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:1.0:1787758110.913691:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758110.913694:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913695:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758110.913697:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758110.913701:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758110.913703:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758110.913704:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758110.913706:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758110.913708:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758110.913709:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11165 00000100:00080000:1.0:1787758110.913723:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758110.913726:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758110.913728:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758110.913730:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758110.913732:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758110.913734:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000800:00000100:0.0:1787758110.913739:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:1.0:1787758110.913739:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758110.913745:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758110.913747:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758110.913752:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787758110.913754:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758110.913757:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758110.913761:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758110.913764:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758110.913768:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758110.913770:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913771:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913773:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758110.913775:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:1.0:1787758110.913778:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758110.913783:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913785:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913786:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758110.913789:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758110.913791:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305838785 02000000:00000001:1.0:1787758110.913792:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913798:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758110.913799:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758110.913800:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913805:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913808:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:1.0:1787758110.913809:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659715584/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758110.913816:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305838785 00000100:00000001:1.0:1787758110.913817:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758110.913821:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758110.913824:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913825:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758110.913834:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758110.913836:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913838:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758110.913845:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758110/real 1787758110] req@ffff8ffb4a535c00 x1874588659715200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758110.913854:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758110.913855:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758110.913860:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758110.913863:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758110.913865:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.913867:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758110.913869:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758110.913874:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659715200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758110.913882:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.913886:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758110.913889:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.913891:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758110.913892:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.913895:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758110.913897:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758110.913900:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758110.913904:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758110.913907:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758110.913912:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659715200:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758110.913917:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758110.913918:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758110.913919:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758110.913921:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758110.913923:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758110.913924:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758110.913926:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715328:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758110.913929:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758110.913930:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758110.913932:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758110.913933:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758110.913936:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913940:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758110.913941:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758110.913945:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758110.913955:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758110.913958:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758110.913960:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758110.913961:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913962:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758110.913964:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913967:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305838093 00000100:00000001:0.0:1787758110.913969:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305838785 00000100:00000001:0.0:1787758110.913971:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305838093 00000100:00000001:0.0:1787758110.913972:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305838785 00000100:00000001:0.0:1787758110.913974:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758110.913975:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.913976:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758110.913979:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758110/real 1787758110] req@ffff8ffb4a537b80 x1874588659715328/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758110.913984:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758110.913984:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758110.913986:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758110.913987:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758110.913988:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.913989:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758110.913990:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758110.913992:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659715328/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758110.913997:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.913998:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758110.914000:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914001:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758110.914002:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914003:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758110.914013:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758110.914015:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758110.914018:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758110.914019:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758110.914021:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715328:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758110.914024:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914025:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758110.914026:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758110.914028:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758110.914031:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758110.914032:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758110.914034:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758110.914037:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758110.914038:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758110.914040:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758110.914040:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914041:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758110.914042:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758110.914043:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758110.914044:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758110.914045:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758110.914046:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758110.914047:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758110.914048:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758110.914049:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758110.914050:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914051:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914057:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758110.914058:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914059:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758110.914060:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758110.914061:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758110.914062:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758110.914063:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758110.914065:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758110.914066:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758110.914067:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758110.914069:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758110.914071:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758110.914072:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758110.914073:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758110.914074:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758110.914075:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914095:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758110.914096:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758110.914099:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758110.914107:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758110.914109:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758110.914111:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758110.914112:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914113:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758110.914115:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914117:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305838090 00000100:00000001:0.0:1787758110.914119:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305838785 00000100:00000001:0.0:1787758110.914120:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305838090 00000100:00000001:0.0:1787758110.914122:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305838785 00000100:00000001:0.0:1787758110.914124:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758110.914125:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914126:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758110.914129:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758110/real 1787758110] req@ffff8ffb4a537800 x1874588659715456/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758110.914133:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758110.914134:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758110.914135:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758110.914137:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758110.914138:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914139:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758110.914140:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758110.914142:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659715456/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758110.914147:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914149:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758110.914150:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914151:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758110.914152:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914153:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758110.914161:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758110.914163:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758110.914164:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758110.914166:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758110.914168:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758110.914170:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758110.914170:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758110.914171:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758110.914173:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758110.914174:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758110.914176:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758110.914178:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758110.914179:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758110.914180:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758110.914181:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758110.914182:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758110.914184:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914186:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758110.914187:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758110.914190:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758110.914195:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758110.914223:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758110.914225:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758110.914226:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914227:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758110.914229:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914231:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305838091 00000100:00000001:0.0:1787758110.914233:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305838785 00000100:00000001:0.0:1787758110.914235:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305838091 00000100:00000001:0.0:1787758110.914236:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305838785 00000100:00000001:0.0:1787758110.914238:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758110.914239:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914240:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758110.914243:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758110/real 1787758110] req@ffff8ffb4a534700 x1874588659715584/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758110.914248:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758110.914249:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758110.914250:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758110.914251:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758110.914252:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914253:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758110.914254:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758110.914257:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659715584/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758110.914261:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914262:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758110.914264:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914265:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758110.914265:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914267:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758110.914276:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758110.914277:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758110.914280:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758110.914281:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758110.914283:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715584:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758110.914286:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914287:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758110.914287:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758110.914288:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758110.914290:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758110.914291:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758110.914291:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758110.914293:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758110.914294:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758110.914295:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758110.914296:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914297:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758110.914298:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758110.914299:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758110.914300:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758110.914300:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758110.914301:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758110.914303:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758110.914304:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758110.914305:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758110.914305:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914307:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758110.914326:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758110.914327:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758110.914328:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758110.914330:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758111.489309:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758111.489315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758111.489317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758111.489321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758111.489325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758111.489328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758111.489331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758111.489333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758111.489336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758111.681438:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758111.681445:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758111.681452:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758111.681454:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758111.681479:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758111.681481:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758111.681484:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758112.513307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758112.513317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758112.513320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758112.513327:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758112.513332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758112.513338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758112.513342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758112.513347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758112.513353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758112.705301:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758112.705310:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758112.705318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758112.705321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758112.705403:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758112.705409:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758112.705414:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758113.537375:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758113.537384:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758113.537386:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758113.537393:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758113.537398:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758113.537404:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758113.537410:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758113.537415:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758113.537419:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758113.729301:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758113.729309:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758113.729317:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758113.729321:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758113.729325:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758113.729329:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758113.729333:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758113.729337:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758113.729340:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758113.729344:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758113.729514:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758113.729527:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758113.729536:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758114.561340:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758114.561350:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758114.561353:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758114.561360:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758114.561365:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758114.561370:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758114.561373:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758114.561377:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758114.561381:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758114.753349:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758114.753358:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758114.753365:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758114.753367:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758114.753522:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758114.753529:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758114.753534:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758115.585360:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758115.585378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758115.585384:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758115.585396:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758115.585404:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758115.585411:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758115.585416:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758115.585422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758115.585428:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758115.777318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758115.777327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758115.777333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758115.777335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758115.777362:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758115.777365:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758115.777369:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758116.033347:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758116.033360:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758116.033364:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758116.033369:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758116.033375:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758116.033385:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11170 00000100:00080000:1.0:1787758116.033399:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758116.033402:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758116.033406:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758116.033411:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758116.033414:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758116.033418:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758116.033422:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758116.033424:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758116.033426:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758116.033432:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758116.033435:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758116.033438:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758116.033442:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758116.033446:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033449:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758116.033452:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758116.033465:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758116.033468:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033472:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758116.033475:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758116.033480:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033495:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758116.033506:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659715712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758116.033520:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758116.033526:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033591:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758116.033596:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758116.033601:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758116.033603:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758116.033606:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758116.033610:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758116.033613:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11170 00000100:00080000:1.0:1787758116.033619:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758116.033623:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758116.033625:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758116.033628:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758116.033629:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758116.033633:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758116.033635:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758116.033638:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758116.033639:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758116.033643:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758116.033646:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758116.033648:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758116.033654:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758116.033657:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033659:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758116.033665:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758116.033669:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033672:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758116.033678:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758116.033680:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033683:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758116.033686:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758116.033689:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033696:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758116.033704:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659715840/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758116.033711:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758116.033720:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758116.033720:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758116.033727:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.033729:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758116.033736:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758116.033737:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.033742:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758116.033745:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758116.033753:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:1.0:1787758116.033754:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758116.033760:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787758116.033761:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758116.033763:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787758116.033771:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033774:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758116.033786:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11170 00000100:00100000:0.0:1787758116.033789:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659715712:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787758116.033791:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758116.033798:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758116.033806:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787758116.033807:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758116.033814:0:178062:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:0.0:1787758116.033817:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:1.0:1787758116.033818:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758116.033826:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787758116.033828:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758116.033829:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758116.033839:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758116.033841:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758116.033842:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758116.033851:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758116.033854:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758116.033856:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758116.033857:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758116.033863:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758116.033868:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033870:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758116.033874:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758116.033878:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033880:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758116.033886:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758116.033890:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787758116.033891:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033899:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758116.033900:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 02000000:00000001:1.0:1787758116.033901:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758116.033907:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.033915:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758116.033918:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758116.033927:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659715968/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758116.033941:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758116.033946:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.033951:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758116.033956:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758116.033964:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758116.033967:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758116.033970:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758116.033976:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000800:00000100:0.0:1787758116.033985:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787758116.033991:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11170 00000100:00080000:1.0:1787758116.034005:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758116.034006:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758116.034013:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758116.034016:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758116.034021:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758116.034023:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034032:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:1.0:1787758116.034036:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758116.034041:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758116.034043:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758116.034050:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305843905 00000020:00000001:1.0:1787758116.034051:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758116.034062:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758116.034065:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758116.034066:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758116.034069:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758116.034080:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758116.034081:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305843905 00000100:00000001:1.0:1787758116.034093:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034095:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787758116.034097:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758116.034104:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758116.034107:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758116.034111:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758116.034115:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034118:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758116.034120:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758116.034126:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758116.034132:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758116/real 1787758116] req@ffff8ffb4a534a80 x1874588659715712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:1.0:1787758116.034133:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.034142:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758116.034154:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758116.034159:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758116.034160:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787758116.034163:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000400:00000001:0.0:1787758116.034167:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:1.0:1787758116.034169:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758116.034176:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:1.0:1787758116.034176:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.034230:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034236:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758116.034242:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034248:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:1.0:1787758116.034252:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659716096/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758116.034253:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758116.034273:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659715712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758116.034332:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758116.034334:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758116.034342:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034347:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758116.034348:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034361:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034367:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758116.034370:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034376:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758116.034382:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758116.034391:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758116.034401:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758116.034407:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758116.034415:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659715712:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758116.034424:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034429:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758116.034431:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758116.034436:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758116.034443:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758116.034447:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758116.034450:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758116.034456:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758116.034460:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758116.034465:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758116.034468:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034473:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034485:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758116.034488:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034493:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758116.034496:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758116.034499:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758116.034502:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758116.034505:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758116.034510:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758116.034514:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758116.034519:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758116.034525:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715840:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758116.034565:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758116.034570:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758116.034573:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758116.034576:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758116.034583:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034594:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758116.034597:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:0.0:1787758116.034606:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758116.034635:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758116.034645:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758116.034650:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758116.034653:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034658:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758116.034663:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034672:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305843214 00000100:00000001:0.0:1787758116.034678:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305843905 00000100:00000001:0.0:1787758116.034684:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305843214 00000100:00000001:0.0:1787758116.034689:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305843905 00000100:00000001:0.0:1787758116.034694:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758116.034698:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034702:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758116.034711:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758116/real 1787758116] req@ffff8ffb4a536300 x1874588659715840/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758116.034731:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758116.034734:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758116.034740:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758116.034743:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758116.034746:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034750:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758116.034753:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758116.034762:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659715840/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758116.034777:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034782:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758116.034787:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034791:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758116.034793:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.034797:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758116.034837:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758116.034845:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758116.034863:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758116.034868:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758116.034875:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715840:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758116.034883:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758116.034886:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758116.034889:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758116.034894:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758116.034898:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758116.034902:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758116.034909:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715968:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758116.034916:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758116.034920:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758116.034924:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758116.034928:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758116.034934:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.034945:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758116.034948:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758116.034955:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758116.034981:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758116.034991:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758116.034996:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758116.034999:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.035003:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758116.035008:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.035015:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305843211 00000100:00000001:0.0:1787758116.035021:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305843905 00000100:00000001:0.0:1787758116.035026:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305843211 00000100:00000001:0.0:1787758116.035031:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305843905 00000100:00000001:0.0:1787758116.035035:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758116.035039:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.035044:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758116.035053:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758116/real 1787758116] req@ffff8ffb4a534700 x1874588659715968/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758116.035071:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758116.035074:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758116.035079:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758116.035083:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758116.035086:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035090:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758116.035093:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758116.035101:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659715968/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758116.035116:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035120:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758116.035126:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035129:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758116.035132:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035135:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758116.035155:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758116.035162:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758116.035167:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758116.035172:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758116.035179:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659715968:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758116.035228:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758116.035232:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758116.035235:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758116.035240:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758116.035243:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758116.035248:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758116.035255:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758116.035266:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758116.035270:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758116.035273:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758116.035277:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758116.035283:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.035293:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758116.035296:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758116.035303:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758116.035326:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758116.035336:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758116.035341:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758116.035343:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.035347:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758116.035353:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.035359:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305843212 00000100:00000001:0.0:1787758116.035365:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305843905 00000100:00000001:0.0:1787758116.035370:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305843212 00000100:00000001:0.0:1787758116.035374:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305843905 00000100:00000001:0.0:1787758116.035379:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758116.035383:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.035387:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758116.035396:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758116/real 1787758116] req@ffff8ffb4a537800 x1874588659716096/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758116.035413:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758116.035416:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758116.035421:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758116.035425:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758116.035428:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035432:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758116.035434:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758116.035442:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659716096/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758116.035470:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035475:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758116.035481:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035485:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758116.035487:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035492:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758116.035513:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758116.035519:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758116.035525:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758116.035530:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758116.035565:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716096:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758116.035573:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035577:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758116.035579:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758116.035582:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758116.035588:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758116.035591:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758116.035594:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758116.035600:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758116.035602:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758116.035607:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758116.035609:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035614:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758116.035616:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758116.035619:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758116.035622:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758116.035624:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758116.035627:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758116.035631:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758116.035634:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758116.035637:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758116.035639:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035642:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758116.035644:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758116.035647:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758116.035650:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758116.035653:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758116.035655:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758116.035658:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758116.035660:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758116.035664:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758116.035666:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035670:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758116.035677:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758116.035680:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758116.035683:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758116.035687:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758116.609342:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758116.609352:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758116.609356:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758116.609362:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758116.609366:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758116.609372:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758116.609375:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758116.609379:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758116.609382:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758116.801320:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758116.801333:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758116.801337:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758116.801344:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758116.801352:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758116.801356:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758116.801360:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758116.801365:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758116.801374:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758116.801377:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758116.801481:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758116.801488:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758116.801494:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758117.633366:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758117.633383:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758117.633387:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758117.633400:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758117.633408:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758117.633416:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758117.633421:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758117.633428:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758117.633433:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758117.825267:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758117.825275:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758117.825281:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758117.825284:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758117.825358:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758117.825363:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758117.825366:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758118.657276:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758118.657286:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758118.657289:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758118.657295:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758118.657300:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758118.657304:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758118.657307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758118.657311:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758118.657314:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758118.849361:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758118.849370:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758118.849378:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758118.849379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758118.849506:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758118.849514:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758118.849519:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758119.681272:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758119.681280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758119.681282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758119.681287:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758119.681290:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758119.681293:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758119.681296:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758119.681299:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758119.681302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758119.873327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758119.873336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758119.873347:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758119.873349:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758119.873409:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758119.873413:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758119.873418:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758120.705433:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758120.705446:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758120.705475:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758120.705484:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758120.705490:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758120.705496:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758120.705500:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758120.705505:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758120.705509:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758120.897327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758120.897336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758120.897344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758120.897347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758120.897473:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758120.897480:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758120.897486:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758121.153358:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758121.153374:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758121.153378:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758121.153384:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758121.153391:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758121.153401:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11175 00000100:00080000:2.0:1787758121.153417:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758121.153424:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758121.153427:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758121.153434:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758121.153438:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758121.153469:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758121.153474:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758121.153478:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758121.153481:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758121.153489:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758121.153494:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758121.153499:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758121.153505:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758121.153510:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.153514:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758121.153519:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758121.153534:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758121.153537:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.153542:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758121.153547:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758121.153553:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.153570:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758121.153586:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659716224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758121.153604:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758121.153611:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.153637:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758121.153643:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758121.153652:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758121.153655:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758121.153659:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758121.153664:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758121.153670:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11175 00000100:00080000:2.0:1787758121.153677:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758121.153681:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758121.153685:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758121.153691:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758121.153695:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758121.153699:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758121.153703:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758121.153707:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758121.153710:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758121.153717:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758121.153721:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758121.153724:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.153734:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787758121.153734:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758121.153744:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758121.153749:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787758121.153749:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758121.153757:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758121.153761:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.153769:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787758121.153770:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787758121.153776:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00100000:2.0:1787758121.153782:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787758121.153783:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787758121.153801:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758121.153806:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659716224:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758121.153812:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.153826:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758121.153832:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758121.153836:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758121.153840:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.153842:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.153860:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758121.153863:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758121.153870:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758121.153874:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758121.153875:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.153896:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.153909:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:2.0:1787758121.153910:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659716352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758121.153915:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:0.0:1787758121.153928:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758121.153958:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758121.153963:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.153970:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758121.153976:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758121.153985:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758121.153988:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758121.153992:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758121.153999:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000800:00000100:0.0:1787758121.154004:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787758121.154013:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11175 00000100:00080000:2.0:1787758121.154026:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758121.154027:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758121.154036:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758121.154038:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758121.154042:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758121.154045:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154053:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:2.0:1787758121.154061:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758121.154067:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758121.154068:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758121.154076:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305849025 00000020:00000001:2.0:1787758121.154077:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758121.154090:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758121.154096:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758121.154098:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758121.154104:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758121.154109:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305849025 00000100:00080000:2.0:1787758121.154119:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758121.154121:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787758121.154137:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758121.154142:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758121.154147:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758121.154151:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758121.154154:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154158:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.154163:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758121.154164:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:2.0:1787758121.154178:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758121.154249:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.154253:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758121.154261:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758121/real 1787758121] req@ffff8ffb47a38700 x1874588659716224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787758121.154269:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758121.154276:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.154291:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758121.154300:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758121.154309:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758121.154314:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758121.154318:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758121.154325:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154326:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758121.154331:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758121.154337:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.154341:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758121.154349:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758121.154351:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659716480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758121.154360:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659716224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758121.154466:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758121.154479:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758121.154480:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.154497:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154500:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758121.154510:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758121.154514:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787758121.154522:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758121.154525:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758121.154530:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758121.154532:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787758121.154539:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758121.154542:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758121.154549:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758121.154551:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758121.154564:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758121.154565:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11175 00000100:00080000:2.0:1787758121.154586:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758121.154588:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758121.154600:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758121.154601:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787758121.154608:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787758121.154616:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659716224:192.168.201.118@tcp:250:kworker.0 00000020:00000001:2.0:1787758121.154622:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758121.154629:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758121.154630:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787758121.154635:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758121.154636:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758121.154640:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758121.154649:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000020:00000001:2.0:1787758121.154651:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:0.0:1787758121.154659:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:2.0:1787758121.154662:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000020:00000001:0.0:1787758121.154665:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758121.154670:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:2.0:1787758121.154670:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758121.154678:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758121.154682:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00080000:2.0:1787758121.154689:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758121.154690:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758121.154695:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758121.154706:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154707:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787758121.154725:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758121.154732:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:2.0:1787758121.154735:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758121.154737:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154750:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787758121.154752:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154755:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758121.154768:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758121.154769:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758121.154773:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758121.154777:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:2.0:1787758121.154779:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787758121.154782:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758121.154788:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:2.0:1787758121.154795:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.154797:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.154816:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758121.154818:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716352:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758121.154831:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758121.154834:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758121.154836:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758121.154838:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758121.154841:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758121.154845:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.154848:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.154855:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758121.154861:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758121.154863:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758121.154867:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 02000000:00000001:2.0:1787758121.154869:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154874:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758121.154890:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758121.154906:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1787758121.154910:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659716608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758121.154915:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758121.154923:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758121.154929:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154938:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758121.154946:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758121.154963:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758121.154964:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305848334 00000100:00000001:0.0:1787758121.154974:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305849025 00000100:00000001:2.0:1787758121.154976:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.154983:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305848334 00000100:00000001:0.0:1787758121.154989:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305849025 00000100:00000001:0.0:1787758121.154994:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758121.154996:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155001:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155006:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758121.155018:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758121/real 1787758121] req@ffff8ffb47a3a300 x1874588659716352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758121.155041:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758121.155044:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758121.155049:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758121.155052:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758121.155053:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155056:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758121.155058:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758121.155063:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659716352/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758121.155072:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155075:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758121.155078:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155080:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758121.155082:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155084:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758121.155106:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758121.155110:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758121.155115:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758121.155118:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758121.155123:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716352:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758121.155128:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758121.155130:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758121.155132:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.155135:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758121.155138:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758121.155140:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758121.155144:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758121.155148:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758121.155151:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758121.155154:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.155156:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758121.155161:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155167:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758121.155169:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787758121.155174:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758121.155223:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758121.155230:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758121.155233:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758121.155235:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155237:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758121.155240:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155245:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305848332 00000100:00000001:0.0:1787758121.155248:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305849025 00000100:00000001:0.0:1787758121.155251:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305848332 00000100:00000001:0.0:1787758121.155255:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305849025 00000100:00000001:0.0:1787758121.155257:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758121.155260:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155262:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758121.155267:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758121/real 1787758121] req@ffff8ffb47a39180 x1874588659716480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758121.155277:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758121.155278:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758121.155281:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758121.155283:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758121.155285:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155287:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758121.155289:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758121.155294:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659716480/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758121.155302:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155305:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758121.155309:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155311:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758121.155313:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155315:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758121.155333:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758121.155337:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758121.155341:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758121.155344:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758121.155348:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758121.155352:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155354:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758121.155356:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758121.155358:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758121.155361:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758121.155363:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758121.155365:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758121.155368:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758121.155370:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758121.155375:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758121.155376:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155379:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758121.155380:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758121.155382:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758121.155384:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758121.155385:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758121.155387:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758121.155389:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758121.155390:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758121.155393:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758121.155395:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155397:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155401:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758121.155403:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155405:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758121.155407:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758121.155409:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758121.155411:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758121.155413:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.155416:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758121.155418:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758121.155421:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758121.155425:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758121.155428:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758121.155430:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758121.155432:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758121.155434:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758121.155438:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155473:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758121.155477:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787758121.155485:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758121.155510:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758121.155519:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758121.155524:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758121.155527:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155531:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758121.155536:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155543:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305848332 00000100:00000001:0.0:1787758121.155548:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305849025 00000100:00000001:0.0:1787758121.155553:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305848332 00000100:00000001:0.0:1787758121.155558:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305849025 00000100:00000001:0.0:1787758121.155563:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758121.155567:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155570:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758121.155579:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758121/real 1787758121] req@ffff8ffb47a3b480 x1874588659716608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758121.155594:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758121.155596:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758121.155601:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758121.155603:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758121.155605:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155608:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758121.155610:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758121.155615:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659716608/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758121.155625:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155628:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758121.155630:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155632:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758121.155634:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155636:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758121.155656:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758121.155660:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758121.155664:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758121.155667:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758121.155672:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716608:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758121.155676:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155678:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758121.155680:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758121.155682:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758121.155684:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758121.155686:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758121.155688:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758121.155691:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758121.155693:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758121.155697:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758121.155698:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155701:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758121.155705:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758121.155707:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758121.155710:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758121.155713:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758121.729255:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758121.729261:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758121.729263:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758121.729279:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758121.729282:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758121.729285:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758121.729287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758121.729289:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758121.729292:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758121.921298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758121.921306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758121.921313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758121.921316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758121.921381:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758121.921403:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758121.921407:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758122.753263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758122.753268:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758122.753270:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758122.753274:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758122.753277:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758122.753280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758122.753282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758122.753283:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758122.753285:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758122.945270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758122.945276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758122.945281:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758122.945283:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758122.945345:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758122.945348:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758122.945350:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758123.777265:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758123.777272:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758123.777274:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758123.777278:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758123.777281:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758123.777285:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758123.777287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758123.777290:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758123.777292:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758123.969321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758123.969329:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758123.969337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758123.969342:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758123.969345:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758123.969349:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758123.969353:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758123.969355:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758123.969358:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758123.969361:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758123.969383:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758123.969385:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758123.969387:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758124.801304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758124.801316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758124.801320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758124.801329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758124.801335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758124.801341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758124.801346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758124.801351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758124.801357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758124.993338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758124.993347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758124.993360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758124.993364:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758124.993480:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758124.993491:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758124.993496:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758125.825422:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758125.825439:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758125.825443:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758125.825455:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758125.825463:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758125.825470:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758125.825476:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758125.825483:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758125.825489:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758126.017287:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758126.017296:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758126.017306:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758126.017310:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758126.017435:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758126.017440:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758126.017444:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758126.273478:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758126.273501:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758126.273506:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758126.273513:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758126.273523:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758126.273537:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11181 00000100:00080000:1.0:1787758126.273560:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758126.273567:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758126.273572:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758126.273581:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758126.273586:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758126.273592:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758126.273596:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758126.273601:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758126.273603:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758126.273610:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758126.273614:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758126.273618:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758126.273624:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758126.273629:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.273632:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758126.273636:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758126.273658:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758126.273661:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.273666:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758126.273671:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758126.273678:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.273699:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758126.273714:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659716736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758126.273730:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758126.273736:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.273769:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758126.273774:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758126.273779:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758126.273782:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758126.273784:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758126.273789:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758126.273792:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11181 00000100:00080000:1.0:1787758126.273797:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758126.273801:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758126.273804:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758126.273809:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758126.273811:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758126.273815:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758126.273820:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758126.273823:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758126.273825:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758126.273830:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758126.273833:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.273841:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787758126.273842:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758126.273856:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787758126.273859:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758126.273861:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758126.273872:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.273873:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758126.273889:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758126.273891:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787758126.273904:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787758126.273906:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:1.0:1787758126.273918:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758126.273920:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758126.273937:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758126.273938:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758126.273963:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758126.273969:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.273972:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758126.273974:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758126.273978:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.273984:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758126.273988:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659716736:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1787758126.273999:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659716864/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758126.274002:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758126.274016:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758126.274026:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758126.274029:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.274033:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758126.274036:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758126.274043:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758126.274049:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758126.274050:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758126.274070:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758126.274072:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758126.274076:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758126.274082:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758126.274088:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11181 00000100:00000001:0.0:1787758126.274089:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758126.274094:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00080000:1.0:1787758126.274094:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758126.274105:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758126.274110:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758126.274111:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758126.274120:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758126.274123:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758126.274127:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758126.274131:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758126.274135:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758126.274140:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758126.274144:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758126.274147:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758126.274149:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758126.274155:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758126.274159:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.274161:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758126.274165:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758126.274169:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.274171:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758126.274179:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758126.274252:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274261:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305854146 02000000:00000001:1.0:1787758126.274273:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274277:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758126.274280:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758126.274284:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758126.274289:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274290:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305854146 00000100:00000001:1.0:1787758126.274304:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274307:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758126.274316:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787758126.274318:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659716992/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758126.274327:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274376:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.274385:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758126.274387:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758126.274395:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274410:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758126.274412:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274419:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00080000:1.0:1787758126.274422:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758126.274439:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787758126.274441:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758126.274450:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758126.274452:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758126.274458:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758126.274465:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:1.0:1787758126.274471:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11181 00000100:00000001:0.0:1787758126.274474:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758126.274484:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758126.274487:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758126.274489:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758126.274494:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758126.274497:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758126.274501:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758126.274510:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758126.274511:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758126.274526:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716864:192.168.201.118@tcp:38:kworker.0 00000020:00000001:1.0:1787758126.274527:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758126.274536:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758126.274541:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:1.0:1787758126.274546:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:0.0:1787758126.274548:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758126.274551:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758126.274558:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:1.0:1787758126.274563:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:0.0:1787758126.274566:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.274578:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758126.274583:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758126.274588:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787758126.274594:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758126.274596:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:1.0:1787758126.274612:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274616:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787758126.274623:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758126.274631:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758126.274637:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.274642:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758126.274645:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758126.274648:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787758126.274649:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.274659:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758126.274666:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758126.274670:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274674:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305853454 00000100:00000001:1.0:1787758126.274675:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758126.274681:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758126.274683:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305854146 02000000:00000001:1.0:1787758126.274687:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274691:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305853454 00000100:00000001:0.0:1787758126.274697:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305854146 00000100:00000001:1.0:1787758126.274700:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274701:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758126.274716:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758126.274718:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659717120/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758126.274738:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758126.274742:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758126.274746:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758126.274748:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274760:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.274764:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758126.274770:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758126.274778:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758126.274784:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758126.274796:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758126.274804:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758126.274809:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758126.274812:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758126.274816:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758126.274825:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274839:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758126.274842:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758126.274850:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758126.274873:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274880:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305853451 00000100:00000001:0.0:1787758126.274885:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305854146 00000100:00000001:0.0:1787758126.274890:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305853451 00000100:00000001:0.0:1787758126.274894:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305854146 00000100:00000001:0.0:1787758126.274898:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758126.274902:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274907:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274911:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758126.274919:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758126.274924:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787758126.274928:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758126.274931:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758126.274933:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758126.274936:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758126.274939:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758126.274944:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758126.274948:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758126.274952:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758126.274959:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758126.274966:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758126.274970:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758126.274973:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758126.274977:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758126.274983:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.274993:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758126.274996:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758126.275003:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758126.275023:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.275030:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305853451 00000100:00000001:0.0:1787758126.275036:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305854146 00000100:00000001:0.0:1787758126.275040:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305853451 00000100:00000001:0.0:1787758126.275044:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305854146 00000100:00000001:0.0:1787758126.275049:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758126.275053:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.275058:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.275063:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758126.275069:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758126.275072:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787758126.275076:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758126.275079:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758126.275082:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758126.275086:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758126.849369:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758126.849378:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758126.849381:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758126.849398:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758126.849404:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758126.849409:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758126.849412:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758126.849415:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758126.849417:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758127.041267:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758127.041277:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758127.041281:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758127.041284:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758127.041289:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758127.041292:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758127.041294:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758127.041297:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758127.041301:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758127.041303:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758127.041360:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758127.041363:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758127.041365:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758127.873313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758127.873383:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758127.873388:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758127.873397:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758127.873404:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758127.873411:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758127.873416:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758127.873421:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758127.873426:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758128.065283:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758128.065292:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758128.065300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758128.065303:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758128.065369:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758128.065374:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758128.065381:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758128.897399:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758128.897408:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758128.897411:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758128.897417:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758128.897422:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758128.897426:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758128.897429:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758128.897433:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758128.897436:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758129.089368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758129.089382:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758129.089393:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758129.089395:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758129.089503:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758129.089511:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758129.089517:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787758129.345528:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787758129.345585:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787758129.345606:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787758129.345624:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758129.345654:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758129.345659:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758129.345664:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787758129.345681:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787758129.345689:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758129.345695:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758129.345698:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758129.345701:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787758129.345708:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787758129.345716:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758129.345720:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758129.345722:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758129.345725:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787758129.345732:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787758129.345755:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758129.345763:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758129.345766:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758129.345770:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758129.345776:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758129.345786:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758129.345791:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758129.345812:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758126/real 1787758129] req@ffff8ffb4a534700 x1874588659717120/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758129.345839:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758129.345843:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758129.345854:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758129.345859:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758129.345863:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.345868:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758129.345872:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758129.345881:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659717120/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758129.345896:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.345903:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758129.345910:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.345914:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758129.345916:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.345922:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758129.345947:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758129.345955:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758129.345964:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758129.345970:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758129.345978:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717120:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758129.345988:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758129.345997:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758126/real 1787758129] req@ffff8ffb4a535c00 x1874588659716864/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758129.346015:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758129.346018:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758129.346024:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758129.346027:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758129.346029:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346033:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758129.346036:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758129.346043:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659716864/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758129.346058:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346062:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758129.346067:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346070:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758129.346073:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346076:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758129.346098:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758129.346104:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758129.346113:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758129.346117:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758129.346124:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716864:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758129.346131:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758129.346138:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758126/real 1787758129] req@ffff8ffb4a537800 x1874588659716992/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758129.346154:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758129.346157:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758129.346162:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758129.346165:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758129.346227:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346233:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758129.346236:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758129.346245:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659716992/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758129.346258:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346262:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758129.346266:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346269:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758129.346271:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346274:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758129.346330:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758129.346336:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758129.346343:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758129.346347:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758129.346354:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659716992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758129.346362:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758129.346369:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758126/real 1787758129] req@ffff8ffb4a537b80 x1874588659716736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758129.346397:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758129.346400:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758129.346407:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758129.346410:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758129.346413:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346417:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758129.346420:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758129.346427:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659716736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758129.346440:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346444:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758129.346449:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346452:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758129.346454:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346458:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758129.346461:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758129.346466:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758129.346472:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758129.346477:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758129.346483:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659716736:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758129.346492:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346496:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758129.346498:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758129.346503:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758129.346512:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758129.346516:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758129.346521:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758129.346528:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758129.346531:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758129.346537:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758129.346539:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346543:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758129.346546:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758129.346548:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758129.346551:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758129.346553:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758129.346555:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758129.346559:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758129.346561:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758129.346565:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758129.346567:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346570:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758129.346573:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758129.346575:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758129.346579:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758129.346581:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758129.346583:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758129.346587:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758129.346589:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758129.346592:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758129.346594:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346597:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758129.346599:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758129.346601:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758129.346605:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758129.346607:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758129.346609:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758129.346612:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758129.346615:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758129.346618:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758129.346620:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346624:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758129.346638:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758129.346642:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758129.346646:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758129.346649:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758129.921268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758129.921275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758129.921277:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758129.921282:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758129.921285:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758129.921318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758129.921322:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758129.921324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758129.921327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758130.113371:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758130.113379:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758130.113385:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758130.113387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758130.113469:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758130.113475:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758130.113480:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758130.945295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758130.945304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758130.945307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758130.945327:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758130.945332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758130.945335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758130.945338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758130.945340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758130.945343:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758131.137388:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758131.137395:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758131.137402:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758131.137403:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758131.137493:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758131.137500:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758131.137506:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758131.393343:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758131.393353:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758131.393356:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758131.393359:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758131.393364:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758131.393372:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11186 00000100:00080000:1.0:1787758131.393383:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758131.393387:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758131.393389:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758131.393393:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758131.393395:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758131.393398:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758131.393401:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758131.393403:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758131.393404:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758131.393408:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758131.393410:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758131.393413:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758131.393416:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758131.393420:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393422:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758131.393424:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758131.393433:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758131.393435:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393438:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758131.393441:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758131.393445:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393456:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758131.393466:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659717248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758131.393476:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758131.393480:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393499:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758131.393503:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758131.393506:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758131.393508:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758131.393510:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758131.393513:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758131.393516:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11186 00000100:00080000:1.0:1787758131.393520:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758131.393522:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758131.393524:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758131.393527:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787758131.393528:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:1.0:1787758131.393529:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758131.393534:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758131.393534:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:1.0:1787758131.393537:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758131.393539:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758131.393541:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758131.393544:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758131.393545:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787758131.393548:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758131.393551:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:3.0:1787758131.393555:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758131.393557:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.393559:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758131.393563:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758131.393564:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758131.393574:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:3.0:1787758131.393574:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659717248:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787758131.393583:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.393584:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758131.393589:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758131.393590:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:3.0:1787758131.393592:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.393596:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00100000:1.0:1787758131.393599:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787758131.393602:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787758131.393613:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393617:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.393619:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758131.393622:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 02000000:00000001:1.0:1787758131.393625:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787758131.393627:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787758131.393631:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393636:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758131.393639:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758131.393642:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393651:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758131.393661:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659717376/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758131.393673:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758131.393680:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758131.393686:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.393690:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758131.393693:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.393699:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787758131.393700:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787758131.393706:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758131.393708:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758131.393712:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758131.393714:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758131.393718:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758131.393722:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393723:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758131.393733:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11186 00000100:00000001:3.0:1787758131.393733:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305859265 00000100:00080000:1.0:1787758131.393743:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758131.393746:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758131.393748:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758131.393749:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758131.393756:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787758131.393758:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305859265 00000020:00000001:1.0:1787758131.393759:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758131.393765:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758131.393770:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:1.0:1787758131.393771:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758131.393779:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758131.393780:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758131.393781:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787758131.393788:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758131.393793:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758131.393795:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758131.393802:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758131.393808:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:3.0:1787758131.393809:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758131/real 1787758131] req@ffff8ffb4a536300 x1874588659717248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787758131.393813:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758131.393818:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393821:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758131.393828:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758131.393836:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393840:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1787758131.393843:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758131.393847:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:1.0:1787758131.393848:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758131.393852:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.393854:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758131.393855:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758131.393857:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758131.393860:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.393866:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758131.393870:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.393871:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.393880:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758131.393884:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758131.393886:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659717504/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758131.393896:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659717248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758131.393903:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758131.393910:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.393916:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758131.393923:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787758131.393934:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758131.393936:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758131.393942:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:3.0:1787758131.393942:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758131.393947:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758131.393949:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758131.393954:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787758131.393955:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758131.393960:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758131.393964:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11186 00000100:00000001:3.0:1787758131.393965:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758131.393971:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758131.393974:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758131.393977:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787758131.393980:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758131.393984:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758131.393987:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000020:00000001:1.0:1787758131.393991:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:3.0:1787758131.393994:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:1.0:1787758131.393996:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758131.394002:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:3.0:1787758131.394003:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659717248:192.168.201.118@tcp:250:kworker.0 00000020:00000001:1.0:1787758131.394006:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758131.394013:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758131.394016:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787758131.394017:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758131.394022:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758131.394028:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758131.394032:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758131.394036:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758131.394039:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.394042:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758131.394044:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758131.394047:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.394050:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758131.394056:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787758131.394059:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787758131.394060:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.394065:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:1.0:1787758131.394066:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758131.394071:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758131.394072:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758131.394074:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.394079:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:1.0:1787758131.394084:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758131.394085:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758131.394090:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00100000:1.0:1787758131.394096:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659717632/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787758131.394098:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758131.394102:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758131.394106:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758131.394110:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758131.394113:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758131.394115:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758131.394121:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.394126:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.394128:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758131.394132:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.394137:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758131.394139:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758131.394141:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758131.394143:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758131.394146:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.394150:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758131.394153:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758131.394156:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758131.394162:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717376:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758131.394166:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758131.394169:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758131.394171:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.394176:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758131.394180:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.394214:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758131.394216:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:3.0:1787758131.394222:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758131.394241:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758131.394248:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758131.394251:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758131.394253:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.394254:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758131.394258:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.394802:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305858574 00000100:00000001:3.0:1787758131.394809:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305859265 00000100:00000001:3.0:1787758131.394812:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305858574 00000100:00000001:3.0:1787758131.394815:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305859265 00000100:00000001:3.0:1787758131.394818:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758131.394821:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.394824:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758131.394832:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758131/real 1787758131] req@ffff8ffb4a534a80 x1874588659717376/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758131.394845:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758131.394846:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758131.394852:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758131.394855:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758131.394857:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.394860:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758131.394863:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758131.394954:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659717376/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758131.394971:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.394974:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758131.394978:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.394980:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758131.394982:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.394985:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758131.395001:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758131.395006:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758131.395024:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758131.395031:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758131.395034:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717376:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758131.395038:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758131.395040:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758131.395041:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.395043:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758131.395046:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758131.395047:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758131.395050:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758131.395053:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758131.395055:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758131.395057:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.395058:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758131.395062:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395069:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758131.395070:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:3.0:1787758131.395074:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758131.395089:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758131.395094:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758131.395096:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758131.395097:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395099:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758131.395102:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395105:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305858571 00000100:00000001:3.0:1787758131.395107:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305859265 00000100:00000001:3.0:1787758131.395109:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305858571 00000100:00000001:3.0:1787758131.395111:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305859265 00000100:00000001:3.0:1787758131.395113:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758131.395114:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395116:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758131.395119:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758131/real 1787758131] req@ffff8ffb4a537b80 x1874588659717504/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758131.395125:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758131.395126:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758131.395128:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758131.395130:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758131.395131:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395132:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758131.395134:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758131.395137:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659717504/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758131.395143:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395144:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758131.395147:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395148:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758131.395149:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395150:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758131.395160:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758131.395162:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758131.395164:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758131.395166:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758131.395169:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758131.395172:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758131.395173:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758131.395174:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.395176:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758131.395177:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758131.395179:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758131.395213:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758131.395218:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758131.395220:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758131.395221:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758131.395223:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758131.395226:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395239:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758131.395240:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787758131.395244:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758131.395288:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758131.395293:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758131.395295:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758131.395297:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395298:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758131.395301:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395304:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305858572 00000100:00000001:3.0:1787758131.395306:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305859265 00000100:00000001:3.0:1787758131.395308:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305858572 00000100:00000001:3.0:1787758131.395310:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305859265 00000100:00000001:3.0:1787758131.395312:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758131.395313:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395315:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758131.395318:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758131/real 1787758131] req@ffff8ffb4a537800 x1874588659717632/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758131.395329:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758131.395330:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758131.395332:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758131.395333:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758131.395334:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395336:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758131.395337:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758131.395340:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659717632/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758131.395345:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395347:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758131.395349:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395350:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758131.395351:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395352:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758131.395362:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758131.395364:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758131.395366:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758131.395368:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758131.395370:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717632:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758131.395373:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395375:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758131.395376:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758131.395377:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758131.395379:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758131.395380:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758131.395381:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758131.395384:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758131.395385:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758131.395387:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758131.395388:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395389:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758131.395390:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758131.395391:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758131.395393:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758131.395394:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758131.395395:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758131.395396:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758131.395397:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758131.395398:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758131.395399:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395400:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758131.395401:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758131.395402:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758131.395403:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758131.395403:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758131.395404:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758131.395406:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758131.395407:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758131.395407:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758131.395408:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395410:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758131.395413:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758131.395415:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758131.395416:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758131.395418:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758131.969394:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758131.969404:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758131.969407:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758131.969414:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758131.969420:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758131.969426:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758131.969429:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758131.969431:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758131.969434:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758132.161404:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758132.161414:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758132.161421:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758132.161423:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758132.161454:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758132.161457:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758132.161461:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758132.993319:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758132.993332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758132.993336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758132.993345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758132.993352:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758132.993357:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758132.993362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758132.993365:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758132.993369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758133.185298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758133.185305:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758133.185312:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758133.185315:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758133.185318:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758133.185321:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758133.185324:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758133.185327:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758133.185329:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758133.185331:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758133.185402:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758133.185408:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758133.185413:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758134.017297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758134.017306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758134.017309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758134.017314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758134.017318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758134.017322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758134.017326:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758134.017328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758134.017332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758134.209384:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758134.209392:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758134.209399:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758134.209401:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758134.209484:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758134.209492:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758134.209499:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758135.041393:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758135.041408:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758135.041411:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758135.041421:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758135.041427:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758135.041433:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758135.041437:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758135.041440:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758135.041445:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758135.233409:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758135.233418:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758135.233425:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758135.233427:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758135.233516:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758135.233522:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758135.233527:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758136.065382:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758136.065395:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758136.065411:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758136.065421:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758136.065427:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758136.065433:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758136.065437:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758136.065441:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758136.065446:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758136.257335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758136.257348:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758136.257353:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758136.257357:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758136.257365:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758136.257368:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758136.257371:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758136.257374:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758136.257378:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758136.257380:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758136.257408:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758136.257411:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758136.257415:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758136.513386:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758136.513398:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758136.513402:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758136.513406:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758136.513412:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758136.513423:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11191 00000100:00080000:2.0:1787758136.513440:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758136.513445:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758136.513449:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758136.513456:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758136.513461:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758136.513468:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758136.513473:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758136.513477:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758136.513480:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758136.513489:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758136.513495:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758136.513501:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758136.513508:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758136.513514:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.513518:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758136.513522:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758136.513537:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758136.513539:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.513543:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758136.513546:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758136.513551:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.513566:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758136.513579:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659717760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758136.513595:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758136.513602:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.513636:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758136.513643:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758136.513650:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758136.513653:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758136.513657:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758136.513663:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758136.513670:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11191 00000100:00080000:2.0:1787758136.513679:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758136.513686:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758136.513689:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758136.513694:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758136.513698:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758136.513701:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758136.513704:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758136.513709:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758136.513713:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758136.513714:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758136.513724:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758136.513729:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.513732:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787758136.513734:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787758136.513748:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787758136.513751:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758136.513755:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787758136.513758:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.513761:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.513764:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758136.513784:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787758136.513786:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:2.0:1787758136.513798:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.513800:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758136.513809:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.513810:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758136.513827:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758136.513834:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758136.513837:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659717760:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758136.513840:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758136.513846:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758136.513852:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.513861:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787758136.513868:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.513872:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758136.513878:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758136.513886:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659717888/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787758136.513887:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758136.513905:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:3.0:1787758136.513906:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.513915:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.513922:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758136.513929:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758136.513934:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758136.513937:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758136.513940:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758136.513942:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758136.513946:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787758136.513948:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00080000:2.0:1787758136.513959:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11191 00000100:00000001:3.0:1787758136.513963:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787758136.513967:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758136.513973:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758136.513975:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758136.513978:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758136.513981:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758136.513983:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758136.513986:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758136.513991:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758136.513994:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758136.514000:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758136.514003:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758136.514005:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758136.514011:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758136.514014:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.514017:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758136.514020:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758136.514024:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.514027:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758136.514032:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758136.514040:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:3.0:1787758136.514041:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758136.514058:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758136.514062:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758136.514065:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.514078:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514081:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:2.0:1787758136.514090:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659718016/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758136.514091:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758136.514098:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514110:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758136.514123:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.514138:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758136.514138:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305864385 00000100:00000001:2.0:1787758136.514157:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514164:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758136.514168:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758136.514257:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758136.514262:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758136.514265:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758136.514269:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758136.514278:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787758136.514280:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305864386 00000100:00080000:2.0:1787758136.514290:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11191 00000100:00000001:3.0:1787758136.514293:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758136.514303:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787758136.514306:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758136.514308:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758136.514324:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758136.514324:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758136.514338:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:3.0:1787758136.514346:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758136/real 1787758136] req@ffff8ffb47a38000 x1874588659717760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787758136.514349:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758136.514355:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758136.514368:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758136.514381:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:3.0:1787758136.514383:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758136.514391:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758136.514393:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758136.514398:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787758136.514402:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:2.0:1787758136.514410:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758136.514411:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758136.514419:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758136.514423:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514425:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758136.514430:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787758136.514441:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:3.0:1787758136.514447:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659717760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787758136.514455:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758136.514460:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787758136.514466:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514469:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758136.514480:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758136.514482:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758136.514484:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758136.514492:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00100000:2.0:1787758136.514495:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787758136.514497:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758136.514501:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00000001:2.0:1787758136.514509:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758136.514511:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:2.0:1787758136.514525:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758136.514532:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:2.0:1787758136.514541:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787758136.514543:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:2.0:1787758136.514551:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758136.514563:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659717760:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758136.514565:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758136.514574:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758136.514586:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514589:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.514596:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758136.514602:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514602:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758136.514609:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758136.514618:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758136.514628:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00100000:2.0:1787758136.514629:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659718144/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000001:3.0:1787758136.514636:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758136.514648:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758136.514654:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758136.514660:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758136.514666:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758136.514671:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514675:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758136.514682:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514686:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.514705:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758136.514710:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514715:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758136.514723:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758136.514725:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758136.514728:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758136.514731:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.514735:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758136.514739:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758136.514742:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758136.514749:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717888:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758136.514754:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758136.514758:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758136.514761:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.514763:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758136.514768:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514779:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758136.514781:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:3.0:1787758136.514788:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758136.514810:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758136.514817:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758136.514820:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758136.514822:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514825:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758136.514829:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514835:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305863694 00000100:00000001:3.0:1787758136.514839:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305864385 00000100:00000001:3.0:1787758136.514842:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305863694 00000100:00000001:3.0:1787758136.514846:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305864386 00000100:00000001:3.0:1787758136.514849:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758136.514851:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514854:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758136.514859:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758136/real 1787758136] req@ffff8ffb47a38380 x1874588659717888/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758136.514870:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758136.514871:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758136.514875:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758136.514877:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758136.514879:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.514881:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758136.514884:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758136.514889:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659717888/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758136.514897:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.514900:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758136.514904:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.514906:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758136.514908:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.514910:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758136.514934:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758136.514938:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758136.514943:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758136.514947:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758136.514951:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659717888:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758136.514956:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758136.514958:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758136.514960:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.514963:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758136.514964:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758136.514967:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758136.514971:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758136.514976:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758136.514978:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758136.514980:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.514982:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758136.514986:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.514992:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758136.514994:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:3.0:1787758136.514999:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758136.515011:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758136.515016:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758136.515019:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758136.515021:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.515023:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758136.515026:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.515031:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305863691 00000100:00000001:3.0:1787758136.515034:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305864385 00000100:00000001:3.0:1787758136.515037:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305863691 00000100:00000001:3.0:1787758136.515040:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305864386 00000100:00000001:3.0:1787758136.515043:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758136.515046:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.515048:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758136.515053:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758136/real 1787758136] req@ffff8ffb47a39880 x1874588659718016/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758136.515062:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758136.515063:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758136.515066:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758136.515067:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758136.515069:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515071:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758136.515072:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758136.515077:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659718016/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758136.515084:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515087:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758136.515090:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515092:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758136.515094:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515096:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758136.515116:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758136.515119:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758136.515124:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758136.515127:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758136.515131:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758136.515135:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758136.515137:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758136.515140:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.515143:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758136.515145:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758136.515148:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758136.515151:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758136.515155:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758136.515157:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758136.515159:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758136.515161:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758136.515164:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.515169:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758136.515170:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:3.0:1787758136.515241:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758136.515266:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758136.515272:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758136.515275:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758136.515277:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.515279:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758136.515283:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.515288:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305863692 00000100:00000001:3.0:1787758136.515292:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305864385 00000100:00000001:3.0:1787758136.515295:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305863692 00000100:00000001:3.0:1787758136.515298:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305864386 00000100:00000001:3.0:1787758136.515301:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758136.515303:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.515306:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758136.515311:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758136/real 1787758136] req@ffff8ffb47a3a680 x1874588659718144/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758136.515320:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758136.515322:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758136.515325:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758136.515327:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758136.515329:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515332:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758136.515334:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758136.515339:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659718144/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758136.515348:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515351:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758136.515353:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515355:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758136.515357:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515359:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758136.515379:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758136.515382:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758136.515385:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758136.515388:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758136.515391:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718144:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758136.515396:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515399:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758136.515401:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758136.515403:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758136.515406:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758136.515408:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758136.515410:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758136.515414:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758136.515416:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758136.515418:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758136.515420:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515422:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758136.515424:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758136.515426:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758136.515428:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758136.515429:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758136.515430:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758136.515432:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758136.515434:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758136.515437:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758136.515440:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515442:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758136.515443:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758136.515445:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758136.515448:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758136.515449:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758136.515451:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758136.515452:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758136.515454:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758136.515457:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758136.515459:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515461:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758136.515465:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758136.515467:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758136.515469:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758136.515472:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758137.089380:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758137.089389:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758137.089392:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758137.089398:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758137.089402:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758137.089406:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758137.089410:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758137.089413:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758137.089435:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758137.281393:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758137.281405:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758137.281417:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758137.281422:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758137.281490:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758137.281498:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758137.281505:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758138.113257:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758138.113263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758138.113265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758138.113269:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758138.113272:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758138.113275:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758138.113277:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758138.113279:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758138.113282:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758138.305284:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758138.305289:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758138.305294:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758138.305296:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758138.305346:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758138.305349:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758138.305352:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758139.137273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758139.137280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758139.137282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758139.137286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758139.137290:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758139.137294:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758139.137295:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758139.137297:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758139.137300:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758139.329287:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758139.329293:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758139.329298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758139.329299:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758139.329359:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758139.329363:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758139.329366:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758140.161302:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758140.161312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758140.161314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758140.161320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758140.161324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758140.161329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758140.161331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758140.161335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758140.161337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758140.353309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758140.353316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758140.353322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758140.353324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758140.353356:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758140.353366:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758140.353370:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758141.185313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758141.185326:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758141.185329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758141.185337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758141.185343:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758141.185349:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758141.185353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758141.185357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758141.185361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758141.377316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758141.377327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758141.377338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758141.377341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758141.377419:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758141.377426:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758141.377431:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758141.633373:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758141.633387:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758141.633390:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758141.633395:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758141.633401:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758141.633410:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11196 00000100:00080000:1.0:1787758141.633435:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758141.633439:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758141.633441:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758141.633447:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758141.633450:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758141.633454:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758141.633457:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758141.633460:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758141.633462:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758141.633468:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758141.633471:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758141.633475:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758141.633479:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758141.633483:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.633486:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758141.633489:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758141.633502:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758141.633504:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.633508:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758141.633512:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758141.633517:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.633532:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758141.633544:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659718272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758141.633558:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758141.633564:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.633595:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758141.633600:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758141.633604:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758141.633607:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758141.633609:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758141.633613:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758141.633617:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11196 00000100:00080000:1.0:1787758141.633622:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758141.633624:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758141.633626:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758141.633630:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758141.633632:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758141.633636:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758141.633640:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758141.633644:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758141.633645:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758141.633650:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758141.633653:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758141.633656:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787758141.633656:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758141.633665:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787758141.633667:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758141.633670:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758141.633677:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.633678:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758141.633685:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758141.633686:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787758141.633696:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:3.0:1787758141.633696:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758141.633700:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:1.0:1787758141.633704:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758141.633709:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.633711:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758141.633722:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:3.0:1787758141.633723:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659718272:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787758141.633731:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.633734:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787758141.633742:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758141.633744:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758141.633749:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758141.633753:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758141.633760:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758141.633766:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758141.633775:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.633783:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758141.633788:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:1.0:1787758141.633792:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.633794:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:1.0:1787758141.633814:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659718400/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758141.633831:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758141.633835:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.633841:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758141.633845:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787758141.633855:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787758141.633856:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758141.633865:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758141.633867:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758141.633883:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758141.633890:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11196 00000100:00000001:3.0:1787758141.633893:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787758141.633898:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758141.633905:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758141.633906:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758141.633909:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787758141.633915:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758141.633916:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758141.633920:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:3.0:1787758141.633927:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:1.0:1787758141.633928:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758141.633936:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758141.633939:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758141.633942:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787758141.633944:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758141.633948:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758141.633956:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.633957:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305869505 02000000:00000001:1.0:1787758141.633962:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758141.633969:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758141.633973:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.633975:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758141.633976:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758141.633985:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787758141.633989:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305869505 00000008:00000001:1.0:1787758141.633992:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.634001:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634002:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787758141.634011:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787758141.634012:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758141.634019:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634021:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.634025:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:3.0:1787758141.634029:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:1.0:1787758141.634030:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758141.634036:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758141.634044:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758141/real 1787758141] req@ffff8ffb4a535c00 x1874588659718272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758141.634047:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758141.634070:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659718528/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:3.0:1787758141.634072:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758141.634077:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758141.634132:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758141.634137:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758141.634139:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634143:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758141.634146:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758141.634156:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659718272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758141.634247:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758141.634249:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758141.634260:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634260:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758141.634273:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758141.634277:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634278:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758141.634283:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758141.634289:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1787758141.634290:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758141.634297:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758141.634305:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758141.634306:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787758141.634317:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758141.634318:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758141.634327:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00080000:1.0:1787758141.634329:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758141.634335:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659718272:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758141.634339:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787758141.634345:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758141.634351:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11196 00000100:00000001:3.0:1787758141.634353:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758141.634359:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758141.634364:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00080000:1.0:1787758141.634366:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:3.0:1787758141.634370:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:1.0:1787758141.634372:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000020:00000001:3.0:1787758141.634376:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:1.0:1787758141.634378:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758141.634385:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000020:00000001:1.0:1787758141.634386:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758141.634393:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758141.634397:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:3.0:1787758141.634398:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000020:00000001:1.0:1787758141.634400:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758141.634407:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:3.0:1787758141.634407:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758141.634415:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758141.634429:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758141.634429:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758141.634438:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758141.634439:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758141.634451:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:3.0:1787758141.634453:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758141.634457:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758141.634464:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.634467:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758141.634469:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000008:00100000:1.0:1787758141.634471:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:3.0:1787758141.634477:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787758141.634478:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.634490:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634491:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758141.634497:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787758141.634498:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787758141.634500:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758141.634505:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634506:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758141.634513:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.634514:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758141.634521:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758141.634522:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758141.634526:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758141.634529:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758141.634535:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.634539:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758141.634543:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718400:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758141.634554:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:1.0:1787758141.634555:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659718656/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1787758141.634571:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758141.634577:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758141.634584:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758141.634590:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:3.0:1787758141.634594:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.634596:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758141.634607:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634611:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758141.634616:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:3.0:1787758141.634623:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758141.634648:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758141.634656:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758141.634660:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758141.634662:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634664:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758141.634668:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634673:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305868814 00000100:00000001:3.0:1787758141.634677:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305869505 00000100:00000001:3.0:1787758141.634681:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305868814 00000100:00000001:3.0:1787758141.634684:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305869505 00000100:00000001:3.0:1787758141.634687:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758141.634689:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634691:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758141.634698:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758141/real 1787758141] req@ffff8ffb4a534700 x1874588659718400/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758141.634708:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758141.634710:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758141.634713:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758141.634715:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758141.634717:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634719:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758141.634721:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758141.634726:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659718400/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758141.634735:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634738:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758141.634741:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634743:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758141.634744:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634746:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758141.634763:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758141.634767:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758141.634771:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758141.634774:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758141.634779:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718400:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758141.634783:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758141.634785:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758141.634787:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758141.634789:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758141.634791:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758141.634794:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758141.634797:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758141.634801:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758141.634804:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758141.634806:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758141.634808:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758141.634812:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634817:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758141.634819:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787758141.634824:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758141.634836:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758141.634842:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758141.634844:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758141.634846:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634848:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758141.634854:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634869:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305868811 00000100:00000001:3.0:1787758141.634873:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305869505 00000100:00000001:3.0:1787758141.634878:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305868811 00000100:00000001:3.0:1787758141.634882:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305869505 00000100:00000001:3.0:1787758141.634885:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758141.634890:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.634894:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758141.634903:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758141/real 1787758141] req@ffff8ffb4a537800 x1874588659718528/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758141.634922:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758141.634924:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758141.634930:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758141.634933:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758141.634936:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634940:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758141.634942:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758141.634950:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659718528/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758141.634963:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634966:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758141.634969:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634972:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758141.634973:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.634976:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758141.634991:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758141.634994:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758141.634998:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758141.635001:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758141.635006:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758141.635010:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635013:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758141.635015:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758141.635017:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758141.635021:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758141.635023:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758141.635025:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758141.635029:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758141.635031:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758141.635034:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758141.635036:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635038:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758141.635040:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758141.635041:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758141.635043:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758141.635045:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758141.635046:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758141.635048:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758141.635049:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758141.635051:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758141.635053:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635055:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635059:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758141.635061:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.635064:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758141.635065:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758141.635067:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758141.635069:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758141.635071:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758141.635073:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758141.635076:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758141.635078:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758141.635109:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758141.635125:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758141.635128:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758141.635130:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758141.635132:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758141.635136:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.635144:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758141.635146:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:3.0:1787758141.635151:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758141.635168:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758141.635174:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758141.635177:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758141.635179:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.635181:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758141.635211:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.635217:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305868812 00000100:00000001:3.0:1787758141.635220:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305869505 00000100:00000001:3.0:1787758141.635223:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305868812 00000100:00000001:3.0:1787758141.635226:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305869505 00000100:00000001:3.0:1787758141.635229:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758141.635231:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.635234:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758141.635239:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758141/real 1787758141] req@ffff8ffb4a537b80 x1874588659718656/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758141.635249:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758141.635251:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758141.635254:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758141.635256:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758141.635258:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635260:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758141.635262:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758141.635267:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659718656/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758141.635275:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635278:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758141.635280:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635282:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758141.635284:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635286:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758141.635301:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758141.635304:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758141.635308:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758141.635311:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758141.635315:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718656:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758141.635322:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635325:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758141.635326:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758141.635329:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758141.635331:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758141.635333:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758141.635335:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758141.635338:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758141.635340:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758141.635342:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758141.635344:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635346:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758141.635350:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758141.635352:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758141.635355:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758141.635357:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758142.209291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758142.209300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758142.209303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758142.209308:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758142.209329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758142.209334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758142.209337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758142.209340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758142.209344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758142.401272:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758142.401288:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758142.401300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758142.401302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758142.401354:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758142.401359:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758142.401363:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758143.233323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758143.233334:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758143.233338:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758143.233347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758143.233353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758143.233360:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758143.233364:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758143.233369:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758143.233374:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758143.425329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758143.425338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758143.425348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758143.425354:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758143.425361:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758143.425368:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758143.425376:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758143.425382:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758143.425388:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758143.425396:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758143.425491:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758143.425497:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758143.425501:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758144.257459:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758144.257471:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758144.257474:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758144.257481:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758144.257487:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758144.257492:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758144.257496:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758144.257500:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758144.257503:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758144.449253:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758144.449259:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758144.449264:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758144.449265:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758144.449284:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758144.449287:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758144.449289:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758145.281275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758145.281283:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758145.281285:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758145.281291:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758145.281294:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758145.281309:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758145.281311:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758145.281314:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758145.281324:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758145.473309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758145.473318:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758145.473326:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758145.473329:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758145.473425:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758145.473429:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758145.473442:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758146.305327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758146.305336:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758146.305339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758146.305347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758146.305351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758146.305356:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758146.305359:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758146.305362:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758146.305366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758146.497324:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758146.497336:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758146.497340:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758146.497344:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758146.497350:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758146.497353:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758146.497355:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758146.497358:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758146.497363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758146.497365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758146.497461:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758146.497469:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758146.497502:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758146.753340:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758146.753352:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758146.753355:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758146.753360:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758146.753365:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758146.753374:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11201 00000100:00080000:2.0:1787758146.753387:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758146.753390:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758146.753393:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758146.753398:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758146.753401:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758146.753405:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758146.753408:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758146.753411:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758146.753413:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758146.753418:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758146.753421:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758146.753424:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758146.753428:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758146.753431:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753434:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758146.753436:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758146.753450:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758146.753452:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753456:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758146.753460:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758146.753465:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753478:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758146.753490:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659718784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758146.753504:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758146.753509:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753529:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758146.753533:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758146.753537:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758146.753539:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758146.753541:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758146.753545:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758146.753547:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11201 00000100:00080000:2.0:1787758146.753551:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758146.753554:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758146.753556:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758146.753558:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758146.753560:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758146.753563:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758146.753565:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758146.753567:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758146.753569:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758146.753572:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758146.753575:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758146.753576:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758146.753583:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758146.753586:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753588:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758146.753595:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758146.753599:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753603:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758146.753609:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758146.753610:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753613:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758146.753615:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758146.753617:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753623:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.753628:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:2.0:1787758146.753630:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659718912/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758146.753643:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758146.753655:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758146.753657:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758146.753664:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.753665:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758146.753680:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753683:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758146.753693:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:3.0:1787758146.753694:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758146.753706:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787758146.753708:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758146.753727:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:3.0:1787758146.753727:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758146.753741:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758146.753753:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:3.0:1787758146.753754:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659718784:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787758146.753761:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11201 00000100:00080000:2.0:1787758146.753765:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758146.753777:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758146.753781:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787758146.753782:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758146.753791:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758146.753793:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:3.0:1787758146.753795:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:2.0:1787758146.753796:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:3.0:1787758146.753803:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758146.753812:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:2.0:1787758146.753816:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758146.753821:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758146.753823:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:3.0:1787758146.753829:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758146.753835:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758146.753842:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758146.753844:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758146.753865:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758146.753869:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753871:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758146.753875:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758146.753878:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.753878:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758146.753890:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:2.0:1787758146.753891:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758146.753910:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787758146.753910:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787758146.753917:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753921:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758146.753923:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758146.753926:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753950:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758146.753959:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659719040/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758146.753969:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758146.753973:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.753977:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758146.753981:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758146.753985:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758146.753987:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758146.753990:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758146.754034:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758146.754038:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11201 00000100:00080000:2.0:1787758146.754067:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758146.754070:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758146.754072:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758146.754075:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758146.754077:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758146.754079:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758146.754083:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758146.754088:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758146.754091:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758146.754096:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758146.754105:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:3.0:1787758146.754108:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:2.0:1787758146.754109:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758146.754118:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758146.754123:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.754125:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758146.754128:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758146.754132:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.754134:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758146.754142:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:3.0:1787758146.754142:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787758146.754154:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.754157:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758146.754160:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1787758146.754161:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787758146.754163:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754168:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.754173:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754176:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:2.0:1787758146.754227:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659719168/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758146.754239:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758146.754242:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758146.754248:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754253:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754263:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305874626 00000100:00000001:3.0:1787758146.754281:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758146.754291:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305874626 00000100:00000001:3.0:1787758146.754296:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758146.754301:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758146.754315:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754318:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758146.754329:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758146/real 1787758146] req@ffff8ffb47a39500 x1874588659718784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758146.754345:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758146.754348:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758146.754354:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758146.754358:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758146.754360:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754363:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758146.754366:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758146.754372:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659718784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758146.754381:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754386:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758146.754390:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754392:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758146.754393:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754396:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758146.754399:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758146.754404:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758146.754409:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758146.754412:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758146.754417:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659718784:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758146.754422:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754425:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758146.754427:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758146.754431:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758146.754435:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758146.754438:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758146.754441:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758146.754449:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758146.754451:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758146.754456:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758146.754458:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754471:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754481:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758146.754484:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754487:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758146.754489:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758146.754491:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758146.754493:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758146.754495:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758146.754498:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758146.754502:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758146.754504:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758146.754508:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718912:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758146.754512:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758146.754515:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758146.754517:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758146.754519:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758146.754522:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754529:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758146.754530:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:3.0:1787758146.754538:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758146.754555:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758146.754561:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758146.754564:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758146.754566:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754568:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758146.754571:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754576:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305873934 00000100:00000001:3.0:1787758146.754580:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305874626 00000100:00000001:3.0:1787758146.754582:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305873934 00000100:00000001:3.0:1787758146.754585:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305874626 00000100:00000001:3.0:1787758146.754588:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758146.754590:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754592:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758146.754597:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758146/real 1787758146] req@ffff8ffb47a38a80 x1874588659718912/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758146.754605:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758146.754607:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758146.754610:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758146.754612:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758146.754613:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754616:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758146.754617:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758146.754621:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659718912/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758146.754629:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754631:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758146.754634:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754636:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758146.754638:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754639:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758146.754680:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758146.754683:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758146.754688:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758146.754691:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758146.754696:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659718912:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758146.754701:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758146.754703:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758146.754705:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758146.754708:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758146.754710:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758146.754712:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758146.754716:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758146.754720:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758146.754722:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758146.754724:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758146.754727:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758146.754731:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754738:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758146.754740:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:3.0:1787758146.754746:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758146.754759:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758146.754764:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758146.754766:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758146.754768:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754771:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758146.754774:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754787:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305873931 00000100:00000001:3.0:1787758146.754791:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305874626 00000100:00000001:3.0:1787758146.754794:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305873931 00000100:00000001:3.0:1787758146.754797:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305874626 00000100:00000001:3.0:1787758146.754800:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758146.754802:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754804:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758146.754810:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758146/real 1787758146] req@ffff8ffb47a3a680 x1874588659719040/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758146.754820:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758146.754822:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758146.754825:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758146.754826:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758146.754828:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754830:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758146.754832:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758146.754837:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659719040/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758146.754846:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754849:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758146.754877:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754878:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758146.754880:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.754881:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758146.754898:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758146.754900:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758146.754904:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758146.754906:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758146.754909:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758146.754913:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758146.754915:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758146.754916:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758146.754930:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758146.754932:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758146.754935:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758146.754938:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758146.754941:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758146.754944:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758146.754946:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758146.754948:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758146.754951:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754956:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758146.754958:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:3.0:1787758146.754963:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758146.754973:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758146.754977:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758146.754979:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758146.754981:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754983:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758146.754986:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.754989:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305873931 00000100:00000001:3.0:1787758146.755015:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305874626 00000100:00000001:3.0:1787758146.755027:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305873931 00000100:00000001:3.0:1787758146.755029:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305874626 00000100:00000001:3.0:1787758146.755032:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758146.755035:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.755036:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758146.755041:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758146/real 1787758146] req@ffff8ffb47a39880 x1874588659719168/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758146.755049:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758146.755051:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758146.755054:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758146.755055:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758146.755057:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755059:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758146.755060:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758146.755065:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659719168/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758146.755072:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755074:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758146.755077:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755079:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758146.755081:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755082:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758146.755100:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758146.755104:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758146.755107:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758146.755110:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758146.755113:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719168:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758146.755117:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755119:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758146.755120:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758146.755122:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758146.755124:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758146.755126:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758146.755127:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758146.755130:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758146.755132:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758146.755136:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758146.755137:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755139:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758146.755140:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758146.755142:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758146.755144:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758146.755145:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758146.755146:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758146.755148:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758146.755149:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758146.755153:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758146.755154:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755156:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758146.755157:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758146.755158:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758146.755160:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758146.755161:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758146.755162:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758146.755164:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758146.755165:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758146.755167:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758146.755169:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755171:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758146.755175:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758146.755177:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758146.755179:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758146.755181:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758147.329410:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758147.329428:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758147.329434:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758147.329446:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758147.329454:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758147.329462:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758147.329468:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758147.329473:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758147.329479:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758147.521318:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758147.521327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758147.521340:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758147.521343:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758147.521404:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758147.521413:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758147.521420:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758148.353392:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758148.353404:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758148.353407:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758148.353416:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758148.353421:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758148.353427:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758148.353431:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758148.353436:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758148.353440:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758148.545300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758148.545307:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758148.545315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758148.545317:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758148.545344:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758148.545347:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758148.545350:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758149.377346:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758149.377359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758149.377362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758149.377372:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758149.377377:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758149.377384:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758149.377388:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758149.377392:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758149.377396:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758149.569305:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758149.569315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758149.569323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758149.569326:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758149.569409:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758149.569414:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758149.569419:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758150.401318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758150.401327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758150.401330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758150.401337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758150.401341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758150.401346:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758150.401350:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758150.401354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758150.401357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758150.593371:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758150.593382:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758150.593392:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758150.593396:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758150.593484:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758150.593490:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758150.593495:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758151.425302:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758151.425311:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758151.425314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758151.425321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758151.425325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758151.425330:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758151.425333:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758151.425337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758151.425340:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758151.617335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758151.617344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758151.617351:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758151.617352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758151.617454:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758151.617460:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758151.617466:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758151.873353:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758151.873366:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758151.873369:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758151.873374:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758151.873380:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758151.873389:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11206 00000100:00080000:3.0:1787758151.873402:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758151.873406:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758151.873409:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758151.873414:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758151.873418:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758151.873422:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758151.873425:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758151.873428:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758151.873430:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758151.873435:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758151.873439:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758151.873443:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758151.873447:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758151.873451:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873454:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758151.873457:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758151.873470:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758151.873473:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873477:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758151.873480:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758151.873485:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873499:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758151.873511:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659719296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758151.873525:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758151.873530:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873556:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758151.873560:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758151.873564:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758151.873566:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758151.873569:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758151.873572:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758151.873575:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11206 00000100:00080000:3.0:1787758151.873579:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758151.873581:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758151.873583:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758151.873586:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758151.873587:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758151.873590:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758151.873593:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758151.873596:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758151.873598:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758151.873603:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758151.873609:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758151.873611:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758151.873627:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787758151.873632:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758151.873636:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787758151.873639:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.873642:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758151.873645:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758151.873659:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758151.873665:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787758151.873666:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787758151.873676:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:3.0:1787758151.873677:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873685:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.873687:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758151.873695:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758151.873695:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758151.873701:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873707:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:0.0:1787758151.873708:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659719296:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787758151.873710:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758151.873717:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.873720:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758151.873726:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.873727:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758151.873734:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.873740:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787758151.873743:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659719424/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758151.873749:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873772:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758151.873776:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873781:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.873785:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787758151.873787:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758151.873792:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00080000:3.0:1787758151.873798:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758151.873802:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758151.873804:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758151.873809:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758151.873813:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758151.873817:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11206 00000100:00080000:3.0:1787758151.873824:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758151.873826:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758151.873828:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758151.873831:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758151.873833:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758151.873837:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758151.873841:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758151.873844:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758151.873846:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758151.873856:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1787758151.873858:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758151.873863:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758151.873869:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758151.873874:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758151.873878:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.873879:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758151.873883:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758151.873889:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00100000:3.0:1787758151.873890:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758151.873897:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787758151.873898:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.873934:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758151.873939:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.873946:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758151.873954:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758151.873956:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305879745 02000000:00000001:3.0:1787758151.873963:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.873971:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758151.873973:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787758151.873975:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758151.873986:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.873987:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305879745 00000100:00000001:0.0:1787758151.874000:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758151.874005:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758151.874006:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874009:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874018:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758151.874027:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659719552/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758151.874032:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758151/real 1787758151] req@ffff8ffb45c33480 x1874588659719296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758151.874091:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:0.0:1787758151.874093:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787758151.874101:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758151.874102:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758151.874110:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758151.874112:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874117:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:3.0:1787758151.874120:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758151.874122:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874131:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787758151.874133:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758151.874137:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758151.874138:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758151.874147:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758151.874154:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659719296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758151.874165:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758151.874177:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11206 00000100:00080000:3.0:1787758151.874224:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758151.874227:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758151.874230:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758151.874234:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758151.874236:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758151.874237:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787758151.874239:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758151.874247:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787758151.874248:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758151.874261:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758151.874262:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758151.874270:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758151.874270:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758151.874277:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758151.874284:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758151.874286:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758151.874293:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758151.874295:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758151.874303:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00080000:0.0:1787758151.874304:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:3.0:1787758151.874308:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758151.874316:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.874321:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00080000:0.0:1787758151.874323:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1787758151.874325:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758151.874336:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000008:00000001:3.0:1787758151.874337:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.874344:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758151.874365:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:0.0:1787758151.874367:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659719296:192.168.201.118@tcp:250:kworker.0 02000000:00000001:3.0:1787758151.874369:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758151.874376:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758151.874381:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758151.874387:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874389:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758151.874395:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874399:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758151.874407:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00100000:3.0:1787758151.874410:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659719680/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758151.874414:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758151.874425:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787758151.874427:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000020:00000001:0.0:1787758151.874433:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:3.0:1787758151.874434:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758151.874439:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787758151.874442:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.874449:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758151.874454:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758151.874458:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758151.874460:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874465:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874475:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758151.874478:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874481:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758151.874483:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758151.874486:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758151.874488:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758151.874490:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.874494:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758151.874497:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758151.874500:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758151.874506:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719424:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758151.874511:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758151.874514:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758151.874516:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.874518:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758151.874523:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874532:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758151.874534:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787758151.874540:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758151.874558:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758151.874565:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758151.874569:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758151.874570:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874573:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758151.874576:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874581:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305879054 00000100:00000001:0.0:1787758151.874585:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305879745 00000100:00000001:0.0:1787758151.874588:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305879054 00000100:00000001:0.0:1787758151.874602:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305879745 00000100:00000001:0.0:1787758151.874605:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758151.874607:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874610:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758151.874615:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758151/real 1787758151] req@ffff8ffb45c33800 x1874588659719424/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758151.874627:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758151.874629:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758151.874632:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758151.874634:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758151.874636:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874638:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758151.874640:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758151.874645:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659719424/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758151.874654:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874657:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758151.874660:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874662:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758151.874663:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874665:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758151.874682:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758151.874685:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758151.874689:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758151.874691:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758151.874695:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719424:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758151.874700:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758151.874702:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758151.874703:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.874706:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758151.874708:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758151.874711:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758151.874714:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758151.874719:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758151.874720:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758151.874723:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.874725:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758151.874729:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874734:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758151.874736:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:0.0:1787758151.874741:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758151.874762:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758151.874767:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758151.874769:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758151.874771:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874773:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758151.874776:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874780:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305879051 00000100:00000001:0.0:1787758151.874783:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305879745 00000100:00000001:0.0:1787758151.874786:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305879051 00000100:00000001:0.0:1787758151.874789:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305879745 00000100:00000001:0.0:1787758151.874792:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758151.874794:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874796:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758151.874801:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758151/real 1787758151] req@ffff8ffb45c31180 x1874588659719552/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758151.874810:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758151.874811:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758151.874814:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758151.874816:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758151.874817:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874819:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758151.874821:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758151.874826:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659719552/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758151.874834:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874837:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758151.874840:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874842:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758151.874844:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.874846:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758151.874860:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758151.874863:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758151.874867:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758151.874870:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758151.874874:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758151.874877:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758151.874879:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758151.874881:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.874883:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758151.874885:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758151.874887:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758151.874891:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719680:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758151.874894:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758151.874896:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758151.874898:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758151.874900:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758151.874932:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874939:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758151.874941:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:0.0:1787758151.874946:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758151.874962:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758151.874967:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758151.874970:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758151.874972:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874974:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758151.874978:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874982:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305879051 00000100:00000001:0.0:1787758151.874985:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305879745 00000100:00000001:0.0:1787758151.874988:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305879051 00000100:00000001:0.0:1787758151.874991:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305879745 00000100:00000001:0.0:1787758151.874994:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758151.874996:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.874999:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758151.875004:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758151/real 1787758151] req@ffff8ffb45c30000 x1874588659719680/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758151.875013:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758151.875015:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758151.875017:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758151.875019:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758151.875022:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875024:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758151.875026:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758151.875030:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659719680/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758151.875038:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875041:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758151.875044:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875045:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758151.875047:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875049:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758151.875063:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758151.875066:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758151.875070:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758151.875073:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758151.875077:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719680:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758151.875081:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875083:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758151.875085:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758151.875087:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758151.875090:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758151.875092:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758151.875094:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758151.875097:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758151.875098:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758151.875101:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758151.875103:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875105:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758151.875106:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758151.875108:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758151.875111:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758151.875113:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758151.875114:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758151.875116:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758151.875117:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758151.875119:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758151.875121:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875122:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758151.875124:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758151.875125:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758151.875127:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758151.875128:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758151.875129:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758151.875131:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758151.875132:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758151.875134:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758151.875135:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875137:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758151.875142:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758151.875143:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758151.875146:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758151.875148:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758152.449299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758152.449308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758152.449311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758152.449318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758152.449322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758152.449326:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758152.449330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758152.449333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758152.449337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758152.641401:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758152.641412:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758152.641422:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758152.641425:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758152.641458:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758152.641463:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758152.641469:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758153.473258:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758153.473265:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758153.473268:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758153.473273:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758153.473277:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758153.473280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758153.473282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758153.473284:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758153.473286:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758153.665255:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758153.665261:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758153.665265:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758153.665268:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758153.665270:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758153.665273:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758153.665275:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758153.665277:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758153.665279:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758153.665280:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758153.665333:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758153.665336:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758153.665338:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758154.497306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758154.497312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758154.497314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758154.497318:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758154.497321:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758154.497324:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758154.497326:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758154.497328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758154.497330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758154.689325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758154.689338:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758154.689348:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758154.689351:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758154.689439:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758154.689444:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758154.689448:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758155.521263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758155.521271:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758155.521273:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758155.521279:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758155.521283:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758155.521287:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758155.521291:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758155.521295:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758155.521298:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758155.713335:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758155.713341:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758155.713346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758155.713347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758155.713429:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758155.713434:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758155.713437:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758156.545413:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758156.545424:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758156.545427:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758156.545435:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758156.545441:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758156.545448:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758156.545453:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758156.545457:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758156.545462:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758156.737311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758156.737318:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758156.737321:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758156.737324:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758156.737328:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758156.737330:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758156.737332:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758156.737334:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758156.737340:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758156.737342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758156.737368:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758156.737370:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758156.737372:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758156.993314:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758156.993322:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758156.993324:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758156.993327:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758156.993330:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758156.993337:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11211 00000100:00080000:2.0:1787758156.993346:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758156.993349:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758156.993351:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758156.993354:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758156.993356:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758156.993358:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758156.993360:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758156.993361:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758156.993362:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758156.993365:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758156.993366:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993368:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758156.993371:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758156.993373:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993375:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758156.993377:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993387:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758156.993389:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993391:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758156.993393:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758156.993397:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993407:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758156.993414:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659719808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758156.993422:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758156.993426:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993440:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758156.993442:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758156.993445:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758156.993446:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758156.993448:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758156.993449:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758156.993451:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11211 00000100:00080000:2.0:1787758156.993453:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758156.993454:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758156.993455:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758156.993457:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758156.993458:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758156.993459:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758156.993461:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758156.993462:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758156.993463:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758156.993465:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758156.993466:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993467:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758156.993471:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758156.993472:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993473:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758156.993477:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758156.993479:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993481:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993483:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758156.993484:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993485:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758156.993486:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758156.993487:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993490:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758156.993493:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659719936/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758156.993499:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758156.993500:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993502:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758156.993504:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758156.993507:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758156.993508:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758156.993509:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758156.993512:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758156.993514:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11211 00000100:00080000:2.0:1787758156.993516:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758156.993517:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758156.993518:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758156.993521:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758156.993523:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787758156.993523:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758156.993526:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758156.993527:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:2.0:1787758156.993528:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758156.993530:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000020:00000001:2.0:1787758156.993531:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758156.993533:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993534:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758156.993537:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758156.993539:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:2.0:1787758156.993541:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787758156.993544:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758156.993544:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993548:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758156.993550:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993552:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758156.993555:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993557:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787758156.993559:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659719808:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1787758156.993560:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758156.993566:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993568:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787758156.993569:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758156.993573:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758156.993573:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758156.993577:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993577:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993579:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758156.993581:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758156.993582:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758156.993588:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993592:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993599:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993603:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:2.0:1787758156.993603:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659720064/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758156.993607:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:0.0:1787758156.993613:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758156.993616:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758156.993619:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993621:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758156.993623:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758156.993626:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758156.993627:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758156.993629:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758156.993631:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758156.993633:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11211 00000100:00080000:2.0:1787758156.993635:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758156.993636:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758156.993637:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758156.993639:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758156.993640:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758156.993641:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758156.993643:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758156.993644:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758156.993645:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758156.993648:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758156.993652:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758156.993653:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:2.0:1787758156.993655:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758156.993658:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758156.993659:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993661:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758156.993662:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758156.993664:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993665:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993667:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787758156.993669:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758156.993671:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993672:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758156.993673:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758156.993675:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758156.993676:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758156.993677:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993679:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758156.993682:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993686:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758156.993688:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659720192/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758156.993691:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305884865 00000100:00000001:0.0:1787758156.993702:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758156.993702:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758156.993704:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758156.993708:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993710:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305884865 00000100:00000001:0.0:1787758156.993716:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758156.993725:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758156.993727:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993730:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758156.993735:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758156/real 1787758156] req@ffff8ffb47a38380 x1874588659719808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758156.993747:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758156.993749:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758156.993753:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758156.993756:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758156.993758:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993761:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758156.993764:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758156.993769:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659719808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758156.993785:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993788:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758156.993791:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993793:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758156.993794:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993797:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758156.993799:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758156.993802:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758156.993806:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758156.993808:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758156.993838:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659719808:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758156.993843:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758156.993844:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758156.993845:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758156.993847:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758156.993848:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758156.993849:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758156.993851:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719936:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758156.993854:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758156.993855:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758156.993856:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758156.993857:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758156.993860:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993865:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758156.993866:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:0.0:1787758156.993870:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758156.993879:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758156.993883:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758156.993885:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758156.993886:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993887:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758156.993890:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993893:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305884173 00000100:00000001:0.0:1787758156.993895:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305884865 00000100:00000001:0.0:1787758156.993896:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305884173 00000100:00000001:0.0:1787758156.993898:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305884865 00000100:00000001:0.0:1787758156.993899:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758156.993901:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993902:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758156.993904:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758156/real 1787758156] req@ffff8ffb47a38000 x1874588659719936/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758156.993909:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758156.993910:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758156.993912:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758156.993913:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758156.993914:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993915:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758156.993916:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758156.993918:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659719936/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758156.993923:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993924:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758156.993926:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993927:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758156.993928:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993929:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758156.993939:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758156.993940:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758156.993942:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758156.993944:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758156.993946:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659719936:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758156.993949:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993951:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758156.993952:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758156.993954:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758156.993956:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758156.993958:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758156.993959:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758156.993962:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758156.993963:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758156.993966:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758156.993967:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993968:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758156.993969:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758156.993970:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758156.993971:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758156.993972:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758156.993972:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758156.993973:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758156.993974:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758156.993975:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758156.993975:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993977:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.993982:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758156.993983:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.993985:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758156.993986:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758156.993986:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758156.993987:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758156.993988:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758156.993989:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758156.993990:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758156.993992:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758156.993994:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758156.993996:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758156.993997:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758156.993998:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758156.993999:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758156.994001:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994004:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758156.994005:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787758156.994008:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758156.994014:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758156.994017:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758156.994019:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758156.994020:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994021:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758156.994022:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994024:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305884170 00000100:00000001:0.0:1787758156.994026:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305884865 00000100:00000001:0.0:1787758156.994027:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305884170 00000100:00000001:0.0:1787758156.994029:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305884865 00000100:00000001:0.0:1787758156.994030:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758156.994032:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994033:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758156.994035:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758156/real 1787758156] req@ffff8ffb47a3b480 x1874588659720064/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758156.994039:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758156.994040:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758156.994041:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758156.994042:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758156.994043:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994044:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758156.994045:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758156.994047:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659720064/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758156.994053:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994055:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758156.994057:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994059:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758156.994060:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994062:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758156.994074:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758156.994077:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758156.994081:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758156.994083:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758156.994087:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758156.994091:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758156.994093:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758156.994094:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758156.994097:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758156.994099:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758156.994101:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758156.994104:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758156.994107:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758156.994109:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758156.994110:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758156.994112:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758156.994114:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994118:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758156.994119:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787758156.994122:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758156.994130:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758156.994134:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758156.994135:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758156.994136:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994137:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758156.994139:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994141:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305884170 00000100:00000001:0.0:1787758156.994143:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305884865 00000100:00000001:0.0:1787758156.994145:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305884170 00000100:00000001:0.0:1787758156.994146:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305884865 00000100:00000001:0.0:1787758156.994149:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758156.994150:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994152:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758156.994154:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758156/real 1787758156] req@ffff8ffb47a39180 x1874588659720192/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758156.994159:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758156.994160:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758156.994162:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758156.994163:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758156.994164:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994165:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758156.994166:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758156.994168:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659720192/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758156.994172:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994174:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758156.994175:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994176:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758156.994177:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994178:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758156.994208:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758156.994210:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758156.994212:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758156.994213:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758156.994215:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720192:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758156.994218:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994219:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758156.994220:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758156.994221:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758156.994223:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758156.994224:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758156.994226:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758156.994227:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758156.994228:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758156.994230:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758156.994231:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994232:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758156.994233:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758156.994233:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758156.994234:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758156.994235:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758156.994236:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758156.994238:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758156.994238:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758156.994240:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758156.994241:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994242:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758156.994244:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758156.994245:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758156.994246:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758156.994248:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758157.569361:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758157.569373:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758157.569376:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758157.569383:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758157.569389:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758157.569394:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758157.569397:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758157.569402:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758157.569406:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758157.761269:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758157.761276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758157.761282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758157.761284:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758157.761359:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758157.761368:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758157.761374:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758158.593278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758158.593284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758158.593286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758158.593290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758158.593293:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758158.593296:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758158.593298:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758158.593300:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758158.593302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758158.785273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758158.785279:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758158.785284:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758158.785285:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758158.785359:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758158.785363:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758158.785366:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758159.617293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758159.617300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758159.617301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758159.617305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758159.617307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758159.617311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758159.617312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758159.617314:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758159.617316:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758159.809302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758159.809308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758159.809312:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758159.809313:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758159.809402:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758159.809410:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758159.809416:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758160.641288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758160.641297:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758160.641299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758160.641317:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758160.641322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758160.641327:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758160.641331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758160.641335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758160.641338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758160.833278:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758160.833285:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758160.833292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758160.833294:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758160.833317:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758160.833320:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758160.833323:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758161.665243:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758161.665249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758161.665251:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758161.665255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758161.665258:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758161.665261:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758161.665263:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758161.665265:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758161.665266:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758161.857292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758161.857297:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758161.857303:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758161.857304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758161.857357:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758161.857361:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758161.857364:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758162.113336:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758162.113350:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758162.113353:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758162.113358:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758162.113364:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758162.113389:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11216 00000100:00080000:1.0:1787758162.113398:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758162.113401:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758162.113403:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758162.113406:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758162.113408:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758162.113410:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758162.113412:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758162.113413:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758162.113414:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758162.113418:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758162.113420:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758162.113422:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758162.113425:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758162.113427:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113429:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758162.113430:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758162.113440:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758162.113441:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113444:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758162.113446:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758162.113449:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113459:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758162.113466:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659720320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758162.113473:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758162.113477:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113490:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758162.113492:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758162.113494:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758162.113495:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758162.113497:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758162.113499:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758162.113501:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11216 00000100:00080000:1.0:1787758162.113503:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758162.113504:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758162.113506:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758162.113507:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758162.113509:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758162.113510:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758162.113512:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758162.113514:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758162.113514:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758162.113517:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758162.113518:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758162.113519:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758162.113522:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758162.113523:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758162.113527:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758162.113527:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758162.113531:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787758162.113531:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113535:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758162.113539:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758162.113540:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787758162.113541:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113546:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.113548:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758162.113549:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758162.113552:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758162.113552:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113557:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758162.113558:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758162.113559:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758162.113563:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758162.113566:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659720320:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758162.113568:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113575:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:1.0:1787758162.113577:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659720448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758162.113580:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758162.113583:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.113588:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758162.113588:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758162.113591:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.113593:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113594:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758162.113600:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758162.113602:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758162.113603:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758162.113604:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758162.113606:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758162.113608:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11216 00000100:00080000:1.0:1787758162.113611:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758162.113612:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758162.113612:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758162.113613:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758162.113616:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000020:00000001:1.0:1787758162.113617:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758162.113619:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758162.113621:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758162.113622:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:1.0:1787758162.113626:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758162.113628:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758162.113629:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758162.113631:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758162.113632:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758162.113633:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758162.113635:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758162.113637:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113638:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758162.113640:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758162.113641:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113642:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758162.113646:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758162.113647:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113649:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758162.113650:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758162.113651:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113665:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758162.113671:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659720576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758162.113672:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758162.113680:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758162.113682:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113684:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113686:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787758162.113686:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758162.113691:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758162.113692:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758162.113693:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758162.113697:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758162.113697:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758162.113699:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758162.113700:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:1.0:1787758162.113704:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11216 00000100:00000001:0.0:1787758162.113708:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758162.113708:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758162.113713:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758162.113714:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305889985 00000100:00000001:1.0:1787758162.113715:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758162.113720:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758162.113720:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758162.113752:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:1.0:1787758162.113758:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758162.113759:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305889985 00000020:00000001:1.0:1787758162.113763:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758162.113766:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758162.113766:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758162.113770:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758162.113772:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758162.113775:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758162.113776:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113779:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113781:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:1.0:1787758162.113782:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758162.113797:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:0.0:1787758162.113800:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758162/real 1787758162] req@ffff8ffb4a534a80 x1874588659720320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787758162.113801:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113805:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758162.113808:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758162.113813:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113817:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758162.113822:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758162.113824:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113828:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758162.113831:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000400:00000001:0.0:1787758162.113833:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:1.0:1787758162.113833:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758162.113837:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758162.113840:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113841:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758162.113845:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758162.113848:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758162.113849:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659720704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758162.113851:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758162.113854:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758162.113860:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:0.0:1787758162.113863:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659720320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758162.113864:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.113868:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113876:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.113879:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758162.113881:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.113883:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758162.113883:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.113885:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758162.113887:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758162.113889:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758162.113892:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758162.113894:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758162.113897:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659720320:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758162.113900:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.113901:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758162.113902:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758162.113904:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758162.113907:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758162.113909:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758162.113910:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758162.113914:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758162.113915:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758162.113917:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758162.113918:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.113920:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.113925:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758162.113927:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113928:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758162.113929:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758162.113931:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758162.113931:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758162.113933:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.113934:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758162.113935:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758162.113937:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758162.113939:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720448:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758162.113941:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758162.113943:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758162.113944:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.113945:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758162.113947:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113951:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758162.113952:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:0.0:1787758162.113956:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758162.113965:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758162.113968:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758162.113970:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758162.113971:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113972:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758162.113974:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113977:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305889293 00000100:00000001:0.0:1787758162.113979:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305889985 00000100:00000001:0.0:1787758162.113981:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305889293 00000100:00000001:0.0:1787758162.113983:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305889985 00000100:00000001:0.0:1787758162.113984:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758162.113985:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.113986:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758162.113989:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758162/real 1787758162] req@ffff8ffb4a536300 x1874588659720448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758162.113999:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758162.114000:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758162.114001:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758162.114003:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758162.114003:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114004:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758162.114005:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758162.114008:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659720448/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758162.114012:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114014:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758162.114016:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114017:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758162.114017:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114018:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758162.114028:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758162.114029:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758162.114031:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758162.114032:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758162.114034:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720448:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758162.114037:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758162.114038:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758162.114039:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.114041:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758162.114042:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758162.114043:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758162.114044:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758162.114047:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758162.114048:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758162.114049:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.114050:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758162.114052:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114055:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758162.114056:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758162.114058:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758162.114064:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758162.114067:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758162.114068:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758162.114069:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114070:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758162.114072:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114074:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305889290 00000100:00000001:0.0:1787758162.114075:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305889985 00000100:00000001:0.0:1787758162.114077:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305889290 00000100:00000001:0.0:1787758162.114079:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305889985 00000100:00000001:0.0:1787758162.114080:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758162.114081:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114082:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758162.114085:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758162/real 1787758162] req@ffff8ffb4a537b80 x1874588659720576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758162.114089:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758162.114090:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758162.114091:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758162.114092:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758162.114093:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114094:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758162.114111:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758162.114115:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659720576/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758162.114123:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114126:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758162.114129:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114131:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758162.114133:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114135:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758162.114146:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758162.114149:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758162.114152:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758162.114155:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758162.114158:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758162.114177:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758162.114179:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758162.114180:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.114183:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758162.114185:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758162.114187:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758162.114190:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758162.114194:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758162.114196:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758162.114198:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758162.114200:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758162.114203:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114209:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758162.114269:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758162.114276:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758162.114297:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758162.114303:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758162.114306:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758162.114308:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114310:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758162.114313:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114318:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305889291 00000100:00000001:0.0:1787758162.114321:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305889985 00000100:00000001:0.0:1787758162.114324:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305889291 00000100:00000001:0.0:1787758162.114326:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305889985 00000100:00000001:0.0:1787758162.114329:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758162.114332:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114334:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758162.114339:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758162/real 1787758162] req@ffff8ffb4a537800 x1874588659720704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758162.114348:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758162.114349:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758162.114352:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758162.114354:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758162.114356:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114358:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758162.114360:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758162.114379:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659720704/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758162.114385:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114387:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758162.114389:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114391:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758162.114392:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114394:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758162.114405:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758162.114409:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758162.114412:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758162.114415:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758162.114419:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720704:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758162.114423:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114425:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758162.114427:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758162.114428:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758162.114431:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758162.114432:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758162.114434:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758162.114436:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758162.114437:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758162.114439:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758162.114440:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114441:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758162.114442:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758162.114443:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758162.114444:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758162.114445:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758162.114445:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758162.114447:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758162.114448:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758162.114449:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758162.114449:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114450:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758162.114451:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758162.114452:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758162.114453:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758162.114454:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758162.114454:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758162.114455:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758162.114456:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758162.114457:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758162.114458:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114459:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758162.114462:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758162.114463:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758162.114464:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758162.114465:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758162.689282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758162.689289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758162.689291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758162.689296:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758162.689299:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758162.689302:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758162.689304:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758162.689306:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758162.689309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758162.881279:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758162.881284:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758162.881290:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758162.881292:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758162.881315:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758162.881318:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758162.881321:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758163.713263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758163.713268:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758163.713270:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758163.713274:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758163.713277:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758163.713280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758163.713282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758163.713284:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758163.713286:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758163.905415:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758163.905433:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758163.905437:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758163.905440:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758163.905443:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758163.905445:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758163.905447:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758163.905448:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758163.905450:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758163.905451:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758163.905570:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758163.905580:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758163.905588:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758164.737322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758164.737345:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758164.737347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758164.737352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758164.737355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758164.737359:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758164.737361:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758164.737363:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758164.737365:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758164.929253:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758164.929259:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758164.929263:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758164.929264:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758164.929281:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758164.929283:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758164.929285:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758165.761243:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758165.761249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758165.761251:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758165.761255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758165.761258:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758165.761261:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758165.761262:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758165.761265:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758165.761267:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758165.953292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758165.953297:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758165.953302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758165.953303:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758165.953406:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758165.953410:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758165.953413:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758166.785288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758166.785295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758166.785297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758166.785303:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758166.785306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758166.785310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758166.785313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758166.785316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758166.785318:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758166.977297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758166.977306:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758166.977309:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758166.977313:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758166.977318:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758166.977321:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758166.977324:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758166.977326:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758166.977330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758166.977331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758166.977396:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758166.977401:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758166.977406:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758167.233410:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758167.233419:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758167.233422:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758167.233426:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758167.233430:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758167.233438:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11221 00000100:00080000:1.0:1787758167.233448:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758167.233451:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758167.233453:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758167.233457:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758167.233459:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758167.233462:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758167.233464:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758167.233466:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758167.233467:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758167.233471:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758167.233473:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758167.233475:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758167.233477:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758167.233480:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233482:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758167.233484:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758167.233492:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758167.233493:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233495:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758167.233498:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758167.233502:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233511:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758167.233520:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659720832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758167.233539:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758167.233543:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233560:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758167.233564:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758167.233568:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758167.233570:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758167.233572:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758167.233576:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758167.233580:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11221 00000100:00000001:2.0:1787758167.233584:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787758167.233586:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758167.233591:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758167.233591:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758167.233597:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758167.233597:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:1.0:1787758167.233602:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758167.233602:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000020:00000001:1.0:1787758167.233604:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787758167.233607:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758167.233608:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758167.233613:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:2.0:1787758167.233614:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758167.233615:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758167.233619:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787758167.233619:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758167.233625:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787758167.233626:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233643:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758167.233646:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00100000:2.0:1787758167.233650:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659720832:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787758167.233653:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758167.233660:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758167.233661:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233665:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:2.0:1787758167.233666:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:1.0:1787758167.233672:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787758167.233687:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787758167.233688:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233693:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758167.233694:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758167.233698:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758167.233700:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233702:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758167.233702:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758167.233705:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758167.233707:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233713:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758167.233718:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659720960/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758167.233722:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758167.233725:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758167.233726:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:1.0:1787758167.233728:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.233733:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758167.233734:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758167.233738:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758167.233741:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758167.233742:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758167.233743:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758167.233745:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758167.233747:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11221 00000100:00080000:1.0:1787758167.233751:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758167.233753:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758167.233754:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758167.233756:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758167.233757:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758167.233758:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758167.233760:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758167.233761:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758167.233763:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758167.233765:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758167.233766:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758167.233768:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758167.233770:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758167.233774:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787758167.233774:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758167.233779:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758167.233781:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758167.233783:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233785:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.233785:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787758167.233789:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758167.233791:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787758167.233792:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233795:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787758167.233795:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758167.233799:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758167.233800:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787758167.233802:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233806:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758167.233810:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659721088/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758167.233814:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233816:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758167.233819:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.233822:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305895105 00000100:00000001:1.0:1787758167.233823:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758167.233828:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758167.233830:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758167.233831:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758167.233832:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758167.233834:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787758167.233834:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758167.233839:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11221 00000100:00000001:2.0:1787758167.233841:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305895105 00000100:00080000:1.0:1787758167.233842:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758167.233845:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758167.233846:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758167.233847:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758167.233851:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758167.233852:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787758167.233853:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758167.233856:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758167.233857:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758167.233861:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758167.233862:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758167.233865:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758167.233867:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787758167.233870:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758167/real 1787758167] req@ffff8ffb4a534700 x1874588659720832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758167.233873:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758167.233878:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758167.233881:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758167.233885:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:2.0:1787758167.233887:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:1.0:1787758167.233888:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787758167.233890:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758167.233891:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758167.233895:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758167.233895:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000008:00000001:1.0:1787758167.233899:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.233899:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758167.233902:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.233903:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.233907:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:1.0:1787758167.233908:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758167.233909:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758167.233911:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233915:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:2.0:1787758167.233916:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659720832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787758167.233917:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758167.233919:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.233925:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.233927:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.233930:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1787758167.233931:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659721216/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758167.233935:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.233939:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758167.233941:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758167.233945:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758167.233946:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758167.233949:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.233950:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758167.233955:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787758167.233956:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758167.233961:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758167.233963:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758167.233967:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659720832:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758167.233981:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.233983:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758167.233984:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758167.233986:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758167.233988:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758167.233990:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758167.233991:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758167.233996:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758167.233997:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758167.233999:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758167.234000:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234002:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234009:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758167.234011:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234013:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758167.234015:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758167.234016:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758167.234017:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758167.234018:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758167.234020:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758167.234021:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758167.234023:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758167.234026:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720960:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758167.234029:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758167.234031:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758167.234032:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758167.234034:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758167.234036:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234041:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758167.234043:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787758167.234047:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758167.234056:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758167.234060:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758167.234062:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758167.234063:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234064:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758167.234066:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234070:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305894413 00000100:00000001:2.0:1787758167.234072:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305895105 00000100:00000001:2.0:1787758167.234074:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305894413 00000100:00000001:2.0:1787758167.234077:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305895105 00000100:00000001:2.0:1787758167.234080:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758167.234082:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234084:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758167.234090:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758167/real 1787758167] req@ffff8ffb4a535c00 x1874588659720960/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758167.234099:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758167.234100:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758167.234104:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758167.234114:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758167.234116:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234118:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758167.234119:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758167.234124:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659720960/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758167.234132:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234134:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758167.234136:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234138:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758167.234139:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234141:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758167.234152:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758167.234154:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758167.234156:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758167.234158:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758167.234160:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659720960:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758167.234163:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758167.234164:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758167.234165:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758167.234167:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758167.234168:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758167.234169:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758167.234171:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758167.234173:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758167.234174:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758167.234175:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758167.234176:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758167.234179:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234203:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758167.234204:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787758167.234207:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758167.234217:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758167.234222:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758167.234224:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758167.234226:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234227:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758167.234230:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234232:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305894411 00000100:00000001:2.0:1787758167.234234:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305895105 00000100:00000001:2.0:1787758167.234236:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305894411 00000100:00000001:2.0:1787758167.234238:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305895105 00000100:00000001:2.0:1787758167.234240:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758167.234241:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234242:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758167.234244:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758167/real 1787758167] req@ffff8ffb4a537800 x1874588659721088/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758167.234249:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758167.234250:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758167.234251:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758167.234252:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758167.234253:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234255:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758167.234256:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758167.234258:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659721088/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758167.234263:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234264:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758167.234266:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234266:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758167.234267:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234268:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758167.234276:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758167.234278:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758167.234279:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758167.234281:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758167.234283:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758167.234285:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758167.234285:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758167.234286:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758167.234288:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758167.234289:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758167.234291:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758167.234292:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758167.234294:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758167.234295:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758167.234296:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758167.234297:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758167.234299:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234301:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758167.234302:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787758167.234305:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758167.234311:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758167.234313:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758167.234314:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758167.234315:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234317:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758167.234318:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234320:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305894411 00000100:00000001:2.0:1787758167.234322:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305895105 00000100:00000001:2.0:1787758167.234323:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305894411 00000100:00000001:2.0:1787758167.234325:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305895105 00000100:00000001:2.0:1787758167.234326:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758167.234327:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234328:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758167.234331:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758167/real 1787758167] req@ffff8ffb4a537b80 x1874588659721216/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758167.234335:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758167.234336:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758167.234337:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758167.234338:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758167.234339:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234340:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758167.234341:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758167.234343:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659721216/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758167.234347:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234348:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758167.234349:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234350:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758167.234351:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234352:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758167.234360:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758167.234361:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758167.234364:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758167.234365:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758167.234367:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721216:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758167.234370:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234371:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758167.234372:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758167.234373:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758167.234374:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758167.234375:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758167.234376:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758167.234378:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758167.234378:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758167.234380:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758167.234381:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234382:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758167.234383:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758167.234384:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758167.234385:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758167.234386:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758167.234387:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758167.234388:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758167.234388:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758167.234389:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758167.234390:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234391:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758167.234392:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758167.234392:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758167.234393:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758167.234394:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758167.234395:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758167.234396:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758167.234396:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758167.234397:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758167.234398:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234399:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758167.234401:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758167.234402:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758167.234403:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758167.234404:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758167.809249:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758167.809257:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758167.809261:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758167.809266:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758167.809269:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758167.809273:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758167.809275:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758167.809278:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758167.809281:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758168.001270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758168.001278:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758168.001286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758168.001287:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758168.001314:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758168.001318:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758168.001322:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758168.833256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758168.833261:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758168.833263:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758168.833267:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758168.833270:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758168.833273:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758168.833276:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758168.833277:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758168.833279:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758169.025274:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758169.025279:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758169.025301:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758169.025304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758169.025329:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758169.025332:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758169.025335:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758169.857295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758169.857306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758169.857308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758169.857314:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758169.857317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758169.857322:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758169.857324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758169.857327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758169.857329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758170.049271:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758170.049276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758170.049280:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758170.049282:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758170.049360:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758170.049363:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758170.049366:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758170.881285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758170.881295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758170.881297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758170.881302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758170.881305:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758170.881309:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758170.881311:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758170.881314:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758170.881317:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758171.073264:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758171.073271:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758171.073278:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758171.073279:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758171.073323:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758171.073327:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758171.073329:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758171.905404:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758171.905415:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758171.905418:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758171.905425:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758171.905431:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758171.905436:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758171.905439:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758171.905441:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758171.905444:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758172.097294:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758172.097302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758172.097308:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758172.097309:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758172.097393:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758172.097397:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758172.097399:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758172.353335:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758172.353343:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758172.353345:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758172.353349:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758172.353352:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758172.353358:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11227 00000100:00080000:2.0:1787758172.353367:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758172.353369:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758172.353371:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758172.353374:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758172.353377:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758172.353379:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758172.353381:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758172.353382:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758172.353383:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758172.353386:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758172.353388:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758172.353390:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758172.353393:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758172.353395:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353396:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758172.353398:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758172.353408:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758172.353410:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353413:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758172.353415:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758172.353418:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353428:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758172.353436:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659721344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758172.353448:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758172.353452:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353469:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758172.353471:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758172.353474:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758172.353475:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758172.353476:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758172.353478:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758172.353480:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11227 00000100:00080000:2.0:1787758172.353482:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758172.353483:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758172.353484:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758172.353485:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758172.353486:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758172.353488:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758172.353490:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758172.353491:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758172.353493:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758172.353495:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758172.353497:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758172.353499:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758172.353503:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758172.353504:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353506:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758172.353508:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:2.0:1787758172.353510:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758172.353513:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:2.0:1787758172.353514:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353517:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758172.353519:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758172.353520:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353521:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758172.353524:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787758172.353524:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758172.353526:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353528:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758172.353531:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353531:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.353536:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:2.0:1787758172.353537:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659721472/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758172.353539:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758172.353557:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758172.353566:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659721344:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758172.353570:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758172.353572:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787758172.353574:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.353578:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787758172.353578:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.353581:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758172.353583:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758172.353585:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:2.0:1787758172.353587:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758172.353589:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:0.0:1787758172.353591:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758172.353591:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758172.353593:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758172.353597:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11227 00000100:00080000:2.0:1787758172.353601:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758172.353602:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758172.353604:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758172.353605:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758172.353606:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758172.353608:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:2.0:1787758172.353608:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758172.353610:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758172.353611:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:2.0:1787758172.353614:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758172.353616:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758172.353617:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758172.353621:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758172.353623:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758172.353624:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758172.353627:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758172.353629:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353630:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758172.353632:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758172.353634:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353635:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758172.353640:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758172.353642:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353643:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758172.353644:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758172.353645:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353649:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758172.353653:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1787758172.353658:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659721600/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758172.353664:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758172.353665:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353666:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758172.353668:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758172.353671:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758172.353672:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:2.0:1787758172.353674:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758172.353676:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353677:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758172.353680:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:2.0:1787758172.353681:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758172.353684:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758172.353687:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758172.353688:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11227 00000100:00000001:0.0:1787758172.353693:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305900225 00000100:00080000:2.0:1787758172.353694:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758172.353698:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758172.353699:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758172.353700:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758172.353701:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758172.353703:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758172.353704:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:2.0:1787758172.353705:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758172.353709:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758172.353710:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758172.353711:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305900225 00000100:00080000:2.0:1787758172.353714:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758172.353717:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353718:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:2.0:1787758172.353721:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758172.353724:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758172.353725:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353726:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758172.353727:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758172.353729:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353730:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758172.353731:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353736:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758172.353738:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758172.353740:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:2.0:1787758172.353741:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353744:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758172.353746:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:0.0:1787758172.353747:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758172/real 1787758172] req@ffff8ffb47a3a300 x1874588659721344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787758172.353748:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353754:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758172.353761:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659721728/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758172.353770:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758172.353774:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758172.353777:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758172.353778:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758172.353781:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758172.353785:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758172.353787:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758172.353788:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353791:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758172.353792:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758172.353797:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659721344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758172.353803:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353806:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758172.353808:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353809:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758172.353810:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353812:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758172.353813:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758172.353815:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758172.353818:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758172.353819:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758172.353822:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659721344:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758172.353826:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353827:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758172.353828:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758172.353830:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758172.353833:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758172.353835:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758172.353836:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758172.353839:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758172.353841:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758172.353843:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758172.353844:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353846:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353851:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758172.353852:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353854:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758172.353855:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758172.353856:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758172.353857:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758172.353859:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.353860:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758172.353861:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758172.353863:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758172.353865:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721472:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758172.353867:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758172.353869:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758172.353870:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.353871:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758172.353873:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353876:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758172.353877:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:0.0:1787758172.353880:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758172.353889:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758172.353892:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758172.353893:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758172.353894:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353895:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758172.353897:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353900:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305899533 00000100:00000001:0.0:1787758172.353902:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305900225 00000100:00000001:0.0:1787758172.353903:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305899533 00000100:00000001:0.0:1787758172.353905:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305900225 00000100:00000001:0.0:1787758172.353906:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758172.353908:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353909:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758172.353912:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758172/real 1787758172] req@ffff8ffb47a38700 x1874588659721472/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758172.353917:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758172.353918:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758172.353920:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758172.353921:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758172.353922:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353923:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758172.353923:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758172.353926:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659721472/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758172.353931:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353932:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758172.353934:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353935:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758172.353935:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.353936:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758172.353946:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758172.353947:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758172.353949:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758172.353950:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758172.353952:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721472:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758172.353955:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758172.353956:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758172.353957:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.353958:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758172.353960:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758172.353961:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758172.353963:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758172.353965:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758172.353967:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758172.353968:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.353970:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758172.353973:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.353978:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758172.353980:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787758172.353984:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758172.353994:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758172.353999:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758172.354001:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758172.354002:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.354004:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758172.354007:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.354010:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305899530 00000100:00000001:0.0:1787758172.354012:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305900225 00000100:00000001:0.0:1787758172.354015:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305899530 00000100:00000001:0.0:1787758172.354018:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305900225 00000100:00000001:0.0:1787758172.354020:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758172.354022:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.354024:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758172.354028:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758172/real 1787758172] req@ffff8ffb47a39180 x1874588659721600/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758172.354036:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758172.354037:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758172.354039:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758172.354041:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758172.354042:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354043:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758172.354044:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758172.354047:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659721600/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758172.354056:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354058:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758172.354059:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354060:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758172.354061:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354062:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758172.354071:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758172.354072:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758172.354074:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758172.354075:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758172.354077:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758172.354080:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758172.354081:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758172.354082:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.354083:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758172.354085:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758172.354086:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758172.354087:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758172.354090:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758172.354091:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758172.354092:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758172.354093:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758172.354095:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.354098:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758172.354099:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787758172.354102:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758172.354110:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758172.354113:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758172.354114:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758172.354115:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.354117:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758172.354119:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.354120:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305899530 00000100:00000001:0.0:1787758172.354122:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305900225 00000100:00000001:0.0:1787758172.354124:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305899530 00000100:00000001:0.0:1787758172.354125:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305900225 00000100:00000001:0.0:1787758172.354126:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758172.354128:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.354129:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758172.354131:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758172/real 1787758172] req@ffff8ffb47a3b480 x1874588659721728/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758172.354135:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758172.354136:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758172.354137:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758172.354138:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758172.354139:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354140:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758172.354141:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758172.354144:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659721728/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758172.354148:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354149:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758172.354150:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354151:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758172.354152:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354153:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758172.354161:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758172.354163:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758172.354164:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758172.354165:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758172.354168:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721728:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758172.354170:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354171:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758172.354172:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758172.354173:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758172.354175:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758172.354175:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758172.354176:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758172.354178:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758172.354179:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758172.354205:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758172.354206:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354208:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758172.354209:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758172.354209:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758172.354212:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758172.354213:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758172.354213:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758172.354215:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758172.354216:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758172.354218:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758172.354219:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354220:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758172.354220:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758172.354221:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758172.354222:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758172.354223:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758172.354224:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758172.354224:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758172.354225:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758172.354226:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758172.354227:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354228:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758172.354231:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758172.354231:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758172.354233:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758172.354234:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758172.929238:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758172.929244:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758172.929246:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758172.929250:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758172.929252:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758172.929255:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758172.929257:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758172.929259:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758172.929261:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758173.121288:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758173.121296:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758173.121303:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758173.121306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758173.121326:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758173.121328:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758173.121330:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758173.953255:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758173.953261:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758173.953263:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758173.953267:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758173.953270:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758173.953273:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758173.953275:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758173.953277:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758173.953279:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758174.145244:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758174.145249:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758174.145254:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758174.145257:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758174.145259:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758174.145262:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758174.145265:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758174.145267:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758174.145269:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758174.145271:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758174.145315:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758174.145320:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758174.145324:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758174.977252:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758174.977257:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758174.977259:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758174.977263:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758174.977266:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758174.977269:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758174.977271:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758174.977273:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758174.977275:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758175.169299:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758175.169304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758175.169308:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758175.169309:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758175.169364:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758175.169367:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758175.169370:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758176.001299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758176.001308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758176.001310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758176.001316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758176.001320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758176.001325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758176.001328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758176.001331:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758176.001334:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758176.193273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758176.193285:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758176.193288:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758176.193292:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758176.193298:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758176.193300:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758176.193302:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758176.193304:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758176.193309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758176.193310:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758176.193377:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758176.193384:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758176.193388:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758177.025306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758177.025314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758177.025316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758177.025321:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758177.025324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758177.025328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758177.025331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758177.025333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758177.025336:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758177.217285:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758177.217293:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758177.217298:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758177.217299:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758177.217319:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758177.217321:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758177.217324:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758177.473282:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758177.473291:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758177.473293:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758177.473297:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758177.473302:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758177.473309:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11232 00000100:00080000:3.0:1787758177.473319:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758177.473322:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758177.473325:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758177.473329:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758177.473331:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758177.473334:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758177.473337:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758177.473340:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758177.473341:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758177.473346:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758177.473349:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758177.473352:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758177.473355:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758177.473358:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473360:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758177.473363:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758177.473372:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758177.473374:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473377:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758177.473380:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758177.473384:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473394:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758177.473403:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659721856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758177.473414:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758177.473419:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473433:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758177.473436:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758177.473439:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758177.473441:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758177.473443:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758177.473446:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758177.473449:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11232 00000100:00080000:3.0:1787758177.473480:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758177.473483:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758177.473485:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758177.473488:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758177.473490:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758177.473492:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758177.473495:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758177.473498:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758177.473499:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758177.473500:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758177.473505:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787758177.473506:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758177.473508:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758177.473510:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473513:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758177.473515:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758177.473516:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473520:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758177.473522:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758177.473523:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758177.473529:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473530:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473535:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787758177.473539:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659721856:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1787758177.473542:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758177.473545:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:3.0:1787758177.473548:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473550:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758177.473553:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473554:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473558:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758177.473559:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758177.473563:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758177.473563:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473568:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758177.473571:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758177.473573:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473579:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473580:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758177.473583:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00100000:3.0:1787758177.473587:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659721984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758177.473588:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758177.473599:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758177.473601:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473605:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758177.473608:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758177.473612:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758177.473614:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758177.473616:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758177.473619:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758177.473623:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11232 00000800:00000100:0.0:1787758177.473626:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787758177.473630:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758177.473634:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758177.473636:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758177.473637:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:3.0:1787758177.473639:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758177.473643:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:3.0:1787758177.473643:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758177.473646:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758177.473647:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473651:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:3.0:1787758177.473652:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758177.473656:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758177.473657:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473658:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758177.473661:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305905345 00000100:00080000:3.0:1787758177.473665:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758177.473669:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758177.473671:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758177.473672:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787758177.473675:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758177.473678:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305905345 02000000:00000001:3.0:1787758177.473679:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473684:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758177.473686:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00100000:3.0:1787758177.473688:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758177.473690:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00000001:3.0:1787758177.473693:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473694:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473698:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758177.473698:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758177.473706:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758177/real 1787758177] req@ffff8ffb45c31500 x1874588659721856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758177.473706:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758177.473710:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473715:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758177.473718:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758177.473722:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758177.473723:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758177.473726:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758177.473728:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473729:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758177.473734:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758177.473736:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758177.473738:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659722112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758177.473739:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758177.473742:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758177.473750:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659721856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758177.473755:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758177.473759:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473765:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758177.473766:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473769:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758177.473771:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758177.473772:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.473777:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758177.473779:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758177.473779:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758177.473782:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758177.473783:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758177.473786:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758177.473786:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758177.473792:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787758177.473793:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758177.473797:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1787758177.473800:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11232 00000100:00000001:0.0:1787758177.473801:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758177.473805:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659721856:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758177.473808:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.473809:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758177.473811:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758177.473814:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758177.473817:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758177.473820:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758177.473821:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00080000:3.0:1787758177.473821:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758177.473826:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758177.473827:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787758177.473829:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758177.473830:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758177.473833:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000020:00000001:3.0:1787758177.473834:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758177.473835:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787758177.473838:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758177.473839:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787758177.473842:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758177.473846:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000020:00000001:3.0:1787758177.473846:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758177.473849:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473849:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758177.473852:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758177.473853:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758177.473856:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758177.473858:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758177.473860:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787758177.473861:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758177.473862:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473866:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758177.473869:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758177.473869:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473874:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758177.473876:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758177.473880:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758177.473883:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473885:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787758177.473889:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721984:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1787758177.473889:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758177.473894:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:3.0:1787758177.473896:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473897:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758177.473900:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473900:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.473906:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758177.473908:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758177.473909:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758177.473912:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473915:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758177.473916:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758177.473917:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 02000000:00000001:3.0:1787758177.473921:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758177.473923:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787758177.473924:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758177.473932:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758177.473936:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1787758177.473941:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659722240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758177.473942:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758177.473945:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758177.473948:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473952:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758177.473955:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473961:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305904653 00000100:00000001:3.0:1787758177.473961:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758177.473965:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473966:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305905345 00000100:00000001:0.0:1787758177.473970:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305904653 00000100:00000001:3.0:1787758177.473972:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473973:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305905345 00000100:00000001:0.0:1787758177.473978:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758177.473979:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.473980:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758177.473985:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758177/real 1787758177] req@ffff8ffb45c32a00 x1874588659721984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758177.473991:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758177.473992:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758177.473994:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758177.473996:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758177.473997:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.473999:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758177.474000:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758177.474003:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659721984/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758177.474008:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474009:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758177.474011:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474012:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758177.474013:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474014:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758177.474024:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758177.474025:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758177.474027:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758177.474029:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758177.474031:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659721984:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758177.474034:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758177.474035:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758177.474036:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.474037:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758177.474038:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758177.474040:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758177.474041:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758177.474044:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758177.474045:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758177.474046:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.474047:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758177.474049:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474052:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758177.474053:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787758177.474055:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758177.474062:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758177.474064:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758177.474066:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758177.474067:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474068:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758177.474070:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474072:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305904650 00000100:00000001:0.0:1787758177.474074:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305905345 00000100:00000001:0.0:1787758177.474076:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305904650 00000100:00000001:0.0:1787758177.474078:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305905345 00000100:00000001:0.0:1787758177.474080:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758177.474082:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474083:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758177.474087:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758177/real 1787758177] req@ffff8ffb45c30700 x1874588659722112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758177.474096:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758177.474097:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758177.474099:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758177.474101:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758177.474102:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474104:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758177.474106:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758177.474109:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659722112/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758177.474117:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474119:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758177.474122:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474123:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758177.474125:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474126:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758177.474138:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758177.474141:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758177.474145:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758177.474147:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758177.474150:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758177.474153:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474154:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758177.474155:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758177.474156:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758177.474158:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758177.474159:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758177.474160:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758177.474162:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758177.474163:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758177.474164:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758177.474165:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474166:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758177.474167:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758177.474168:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758177.474168:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758177.474169:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758177.474170:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758177.474171:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758177.474171:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758177.474172:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758177.474173:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474174:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474176:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758177.474177:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474178:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758177.474179:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758177.474180:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758177.474181:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758177.474204:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.474206:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758177.474207:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758177.474208:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758177.474211:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758177.474213:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758177.474214:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758177.474215:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758177.474217:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758177.474220:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474224:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758177.474225:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:0.0:1787758177.474228:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758177.474237:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758177.474240:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758177.474242:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758177.474243:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474244:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758177.474246:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474248:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305904651 00000100:00000001:0.0:1787758177.474250:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305905345 00000100:00000001:0.0:1787758177.474251:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305904651 00000100:00000001:0.0:1787758177.474253:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305905345 00000100:00000001:0.0:1787758177.474255:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758177.474256:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474257:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758177.474259:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758177/real 1787758177] req@ffff8ffb45c30a80 x1874588659722240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758177.474264:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758177.474265:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758177.474266:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758177.474267:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758177.474268:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474270:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758177.474271:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758177.474273:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659722240/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758177.474277:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474279:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758177.474281:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474282:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758177.474283:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474284:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758177.474294:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758177.474295:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758177.474298:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758177.474299:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758177.474301:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722240:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758177.474303:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474304:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758177.474305:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758177.474306:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758177.474307:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758177.474308:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758177.474309:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758177.474310:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758177.474310:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758177.474311:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758177.474312:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474314:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758177.474316:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758177.474317:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758177.474318:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758177.474319:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758178.049289:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758178.049298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758178.049300:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758178.049305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758178.049308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758178.049313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758178.049315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758178.049318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758178.049320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758178.241257:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758178.241263:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758178.241268:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758178.241270:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758178.241311:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758178.241315:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758178.241320:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758179.073278:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758179.073287:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758179.073289:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758179.073295:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758179.073298:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758179.073302:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758179.073304:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758179.073307:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758179.073309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758179.265250:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758179.265256:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758179.265261:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758179.265263:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758179.265307:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758179.265311:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758179.265314:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758180.097268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758180.097273:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758180.097275:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758180.097278:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758180.097281:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758180.097284:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758180.097286:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758180.097288:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758180.097290:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758180.289310:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758180.289317:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758180.289323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758180.289324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758180.289377:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758180.289383:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758180.289386:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758181.121239:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758181.121245:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758181.121248:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758181.121252:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758181.121255:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758181.121258:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758181.121260:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758181.121262:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758181.121264:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758181.313268:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758181.313274:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758181.313279:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758181.313280:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758181.313296:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758181.313298:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758181.313300:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758182.145231:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758182.145237:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758182.145239:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758182.145243:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758182.145246:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758182.145249:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758182.145251:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758182.145254:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758182.145256:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758182.337292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758182.337298:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758182.337303:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758182.337304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758182.337392:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758182.337396:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758182.337399:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758182.593270:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758182.593282:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758182.593284:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758182.593288:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758182.593293:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758182.593300:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11237 00000100:00080000:3.0:1787758182.593311:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758182.593314:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758182.593316:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758182.593321:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758182.593324:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758182.593327:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758182.593329:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758182.593332:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758182.593333:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758182.593338:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758182.593341:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758182.593345:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758182.593348:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758182.593360:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593375:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758182.593378:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758182.593389:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758182.593392:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593395:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758182.593398:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758182.593402:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593414:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758182.593424:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659722368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758182.593435:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758182.593439:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593456:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758182.593459:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758182.593463:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758182.593465:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758182.593466:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758182.593470:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758182.593472:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11237 00000100:00080000:3.0:1787758182.593477:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758182.593480:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758182.593482:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758182.593485:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758182.593486:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758182.593489:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758182.593491:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758182.593494:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758182.593496:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758182.593499:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787758182.593500:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758182.593505:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758182.593506:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593508:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758182.593512:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758182.593513:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758182.593517:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.593521:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758182.593521:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758182.593525:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758182.593526:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.593528:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593532:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:0.0:1787758182.593536:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659722368:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1787758182.593540:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758182.593541:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758182.593547:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00000001:3.0:1787758182.593547:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.593551:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593552:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.593555:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758182.593560:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758182.593561:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758182.593563:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593567:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758182.593569:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758182.593572:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593576:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758182.593579:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:3.0:1787758182.593579:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593586:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:3.0:1787758182.593591:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659722496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758182.593610:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758182.593613:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593618:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758182.593622:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787758182.593625:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758182.593630:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758182.593632:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758182.593634:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758182.593635:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758182.593639:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758182.593641:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758182.593644:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758182.593646:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11237 00000100:00000001:0.0:1787758182.593647:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758182.593654:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758182.593656:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758182.593658:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305910465 00000100:00000001:3.0:1787758182.593662:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758182.593666:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758182.593668:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787758182.593669:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758182.593674:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758182.593676:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305910465 00000020:00000001:3.0:1787758182.593677:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758182.593683:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758182.593685:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758182.593687:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758182.593689:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758182.593691:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758182.593692:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593696:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:3.0:1787758182.593700:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1787758182.593704:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758182/real 1787758182] req@ffff8ffb45c33b80 x1874588659722368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758182.593705:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758182.593711:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758182.593716:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:0.0:1787758182.593719:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787758182.593721:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758182.593722:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758182.593726:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758182.593726:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758182.593730:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758182.593731:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758182.593733:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.593739:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00000001:3.0:1787758182.593739:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593742:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758182.593744:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758182.593747:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659722368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758182.593752:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758182.593755:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758182.593756:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593759:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758182.593760:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758182.593762:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758182.593764:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758182.593765:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758182.593769:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758182.593770:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593771:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758182.593774:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758182.593777:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758182.593780:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758182.593782:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758182.593785:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659722368:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758182.593787:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593788:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.593794:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758182.593796:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758182.593798:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00100000:3.0:1787758182.593801:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659722624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758182.593802:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758182.593806:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758182.593809:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758182.593812:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758182.593814:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758182.593817:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758182.593818:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758182.593821:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.593824:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758182.593824:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593831:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593834:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:3.0:1787758182.593836:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758182.593837:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593842:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758182.593846:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787758182.593846:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758182.593848:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758182.593850:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758182.593851:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758182.593854:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758182.593856:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:0.0:1787758182.593857:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758182.593860:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758182.593863:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593864:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:0.0:1787758182.593868:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722496:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1787758182.593872:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11237 00000100:00000001:0.0:1787758182.593873:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758182.593876:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758182.593879:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.593881:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787758182.593882:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758182.593884:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593888:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758182.593891:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758182.593892:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758182.593894:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000020:00000001:3.0:1787758182.593898:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758182.593900:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787758182.593902:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758182.593907:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758182.593911:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000800:00000100:0.0:1787758182.593913:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758182.593915:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758182.593918:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758182.593919:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758182.593922:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758182.593925:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758182.593929:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758182.593930:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758182.593933:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593936:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593938:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305909773 02000000:00000001:3.0:1787758182.593941:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758182.593943:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305910465 00000100:00000001:0.0:1787758182.593947:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305909773 02000000:00000001:3.0:1787758182.593948:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758182.593949:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305910465 00000100:00000001:0.0:1787758182.593956:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758182.593956:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593963:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593963:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758182.593967:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:3.0:1787758182.593969:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758182.593973:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758182/real 1787758182] req@ffff8ffb45c31c00 x1874588659722496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:3.0:1787758182.593974:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.593980:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758182.593987:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787758182.593988:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:0.0:1787758182.593989:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787758182.593993:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.593994:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758182.593996:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758182.593999:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758182.593999:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758182.594003:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787758182.594003:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758182.594008:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758182.594009:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758182.594011:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659722496/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758182.594017:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594018:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594023:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758182.594026:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758182.594029:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659722752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758182.594030:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758182.594033:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594038:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758182.594046:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758182.594049:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758182.594055:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594057:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758182.594063:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758182.594068:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758182.594071:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758182.594076:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722496:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758182.594082:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758182.594084:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758182.594085:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.594087:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758182.594089:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758182.594090:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758182.594093:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758182.594095:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758182.594097:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758182.594098:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.594099:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758182.594101:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594106:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758182.594107:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:0.0:1787758182.594110:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758182.594119:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758182.594122:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758182.594123:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758182.594124:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594126:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758182.594137:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594139:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305909770 00000100:00000001:0.0:1787758182.594141:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305910465 00000100:00000001:0.0:1787758182.594143:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305909770 00000100:00000001:0.0:1787758182.594144:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305910465 00000100:00000001:0.0:1787758182.594146:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758182.594147:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594148:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758182.594151:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758182/real 1787758182] req@ffff8ffb45c30e00 x1874588659722624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758182.594156:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758182.594156:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758182.594158:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758182.594159:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758182.594160:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594161:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758182.594162:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758182.594164:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659722624/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758182.594173:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594175:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758182.594177:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594178:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758182.594178:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594180:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758182.594221:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758182.594224:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758182.594226:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758182.594227:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758182.594229:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758182.594232:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594233:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758182.594234:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758182.594235:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758182.594237:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758182.594238:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758182.594239:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758182.594240:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758182.594241:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758182.594243:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758182.594244:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594245:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758182.594246:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758182.594246:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758182.594247:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758182.594248:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758182.594249:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758182.594250:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758182.594251:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758182.594252:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758182.594252:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594253:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594256:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758182.594257:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594259:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758182.594260:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758182.594261:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758182.594262:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758182.594263:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.594264:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758182.594265:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758182.594267:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758182.594269:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758182.594271:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758182.594272:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758182.594273:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758182.594274:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758182.594276:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594280:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758182.594281:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:0.0:1787758182.594283:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758182.594291:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758182.594294:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758182.594296:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758182.594297:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594298:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758182.594299:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594301:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305909771 00000100:00000001:0.0:1787758182.594303:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305910465 00000100:00000001:0.0:1787758182.594305:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305909771 00000100:00000001:0.0:1787758182.594306:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305910465 00000100:00000001:0.0:1787758182.594308:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758182.594309:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594310:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758182.594313:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758182/real 1787758182] req@ffff8ffb45c33b80 x1874588659722752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758182.594322:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758182.594323:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758182.594324:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758182.594325:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758182.594326:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594327:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758182.594328:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758182.594330:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659722752/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758182.594343:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594345:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758182.594347:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594348:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758182.594349:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594350:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758182.594359:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758182.594360:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758182.594376:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758182.594378:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758182.594380:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659722752:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758182.594383:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594384:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758182.594385:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758182.594386:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758182.594388:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758182.594389:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758182.594390:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758182.594391:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758182.594392:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758182.594394:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758182.594395:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594396:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758182.594398:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758182.594399:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758182.594400:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758182.594402:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758183.169254:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758183.169261:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758183.169262:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758183.169267:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758183.169270:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758183.169272:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758183.169274:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758183.169276:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758183.169278:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758183.361281:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758183.361286:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758183.361291:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758183.361293:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758183.361321:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758183.361325:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758183.361327:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758184.193260:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758184.193267:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758184.193270:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758184.193276:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758184.193280:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758184.193284:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758184.193286:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758184.193289:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758184.193292:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758184.385282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758184.385287:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758184.385293:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758184.385297:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758184.385300:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758184.385304:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758184.385308:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758184.385310:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758184.385313:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758184.385316:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758184.385379:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758184.385383:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758184.385385:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758185.217282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758185.217289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758185.217292:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758185.217297:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758185.217301:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758185.217305:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758185.217308:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758185.217310:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758185.217313:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758185.409270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758185.409276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758185.409282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758185.409283:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758185.409302:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758185.409305:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758185.409308:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758186.241288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758186.241319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758186.241321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758186.241326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758186.241329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758186.241333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758186.241335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758186.241338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758186.241341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758186.433275:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758186.433283:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758186.433287:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758186.433290:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758186.433306:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758186.433308:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758186.433311:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758186.433314:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758186.433319:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758186.433322:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758186.433366:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758186.433370:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758186.433373:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758187.265305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758187.265319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758187.265321:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758187.265326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758187.265330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758187.265336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758187.265339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758187.265343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758187.265347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758187.457265:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758187.457272:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758187.457307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758187.457308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758187.457362:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758187.457365:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758187.457368:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758187.713306:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758187.713314:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758187.713316:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758187.713319:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758187.713323:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758187.713329:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11242 00000100:00080000:3.0:1787758187.713338:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758187.713341:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758187.713342:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758187.713345:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758187.713348:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758187.713350:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758187.713352:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758187.713353:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758187.713354:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758187.713358:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758187.713360:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758187.713361:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758187.713364:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758187.713366:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713368:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758187.713377:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758187.713387:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758187.713388:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713390:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758187.713392:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758187.713395:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713406:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758187.713413:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659722880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758187.713421:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758187.713425:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713437:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758187.713439:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758187.713443:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758187.713444:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758187.713446:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758187.713448:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758187.713449:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11242 00000100:00080000:3.0:1787758187.713451:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758187.713452:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758187.713453:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758187.713455:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758187.713456:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758187.713458:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758187.713459:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758187.713461:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758187.713463:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758187.713465:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758187.713467:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758187.713468:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758187.713472:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758187.713474:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713475:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758187.713480:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787758187.713480:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758187.713485:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713486:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758187.713487:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713489:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758187.713492:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758187.713493:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758187.713494:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713496:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713497:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758187.713501:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758187.713503:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758187.713503:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.713507:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758187.713509:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.713510:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758187.713520:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659722880:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787758187.713520:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659723008/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758187.713528:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758187.713532:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758187.713536:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713540:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758187.713541:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758187.713544:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713547:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.713549:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758187.713550:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758187.713554:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758187.713555:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758187.713557:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758187.713559:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758187.713561:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11242 00000100:00080000:3.0:1787758187.713563:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758187.713565:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758187.713566:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758187.713566:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758187.713570:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000020:00000001:3.0:1787758187.713571:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758187.713573:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758187.713575:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758187.713576:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:3.0:1787758187.713578:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758187.713580:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758187.713582:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758187.713584:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758187.713586:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758187.713587:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758187.713589:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758187.713591:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713592:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758187.713594:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758187.713595:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713596:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758187.713600:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758187.713601:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713603:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758187.713604:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758187.713605:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713609:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758187.713613:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1787758187.713615:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659723136/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758187.713622:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758187.713624:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713625:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758187.713626:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758187.713630:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758187.713632:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:3.0:1787758187.713633:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758187.713635:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713636:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758187.713639:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758187.713640:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758187.713643:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758187.713647:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758187.713647:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11242 00000100:00000001:0.0:1787758187.713653:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305915585 00000100:00080000:3.0:1787758187.713653:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758187.713658:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758187.713660:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758187.713661:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758187.713662:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758187.713664:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758187.713665:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:3.0:1787758187.713666:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758187.713670:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758187.713671:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758187.713674:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305915585 00000100:00080000:3.0:1787758187.713675:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758187.713680:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713682:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787758187.713682:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758187.713686:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758187.713687:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758187.713689:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713691:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713692:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758187.713695:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758187.713696:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00000001:3.0:1787758187.713700:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758187.713704:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758187/real 1787758187] req@ffff8ffb45c30a80 x1874588659722880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758187.713704:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758187.713707:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758187.713710:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713713:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758187.713716:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758187.713718:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758187.713723:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758187.713732:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659723264/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787758187.713734:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758187.713737:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758187.713741:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758187.713742:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758187.713745:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713746:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758187.713749:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713750:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713754:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758187.713756:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758187.713761:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659722880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758187.713766:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713769:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758187.713771:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713773:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758187.713774:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713776:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758187.713777:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758187.713779:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758187.713782:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758187.713784:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758187.713787:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659722880:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758187.713790:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713791:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758187.713792:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758187.713794:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758187.713797:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758187.713798:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758187.713799:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758187.713802:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758187.713804:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758187.713805:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758187.713806:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713808:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713815:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758187.713816:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713818:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758187.713819:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758187.713820:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758187.713821:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758187.713822:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.713824:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758187.713825:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758187.713826:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758187.713828:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723008:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758187.713830:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758187.713832:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758187.713833:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.713834:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758187.713836:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713839:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758187.713840:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:0.0:1787758187.713843:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758187.713851:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758187.713853:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758187.713855:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758187.713856:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713857:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758187.713859:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713862:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305914893 00000100:00000001:0.0:1787758187.713864:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305915585 00000100:00000001:0.0:1787758187.713865:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305914893 00000100:00000001:0.0:1787758187.713867:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305915585 00000100:00000001:0.0:1787758187.713869:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758187.713870:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713871:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758187.713873:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758187/real 1787758187] req@ffff8ffb45c30700 x1874588659723008/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758187.713884:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758187.713885:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758187.713886:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758187.713887:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758187.713888:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713889:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758187.713890:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758187.713892:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659723008/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758187.713896:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713898:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758187.713900:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713901:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758187.713902:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.713903:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758187.713912:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758187.713914:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758187.713915:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758187.713917:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758187.713919:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723008:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758187.713921:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758187.713922:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758187.713924:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.713925:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758187.713926:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758187.713927:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758187.713929:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758187.713931:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758187.713932:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758187.713934:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.713935:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758187.713938:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713943:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758187.713944:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:0.0:1787758187.713948:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758187.713958:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758187.713963:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758187.713965:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758187.713967:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713968:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758187.713971:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713974:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305914890 00000100:00000001:0.0:1787758187.713977:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305915585 00000100:00000001:0.0:1787758187.713980:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305914890 00000100:00000001:0.0:1787758187.713982:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305915585 00000100:00000001:0.0:1787758187.713984:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758187.713986:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.713988:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758187.713992:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758187/real 1787758187] req@ffff8ffb45c32a00 x1874588659723136/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758187.714001:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758187.714002:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758187.714005:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758187.714008:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758187.714009:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714011:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758187.714013:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758187.714017:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659723136/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758187.714025:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714027:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758187.714031:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714033:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758187.714034:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714036:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758187.714045:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758187.714047:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758187.714049:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758187.714051:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758187.714053:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758187.714056:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758187.714057:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758187.714058:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.714059:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758187.714061:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758187.714062:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758187.714064:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758187.714066:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758187.714068:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758187.714069:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758187.714070:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758187.714073:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.714076:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758187.714077:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:0.0:1787758187.714080:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758187.714089:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758187.714093:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758187.714095:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758187.714097:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.714098:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758187.714101:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.714105:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305914890 00000100:00000001:0.0:1787758187.714107:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305915585 00000100:00000001:0.0:1787758187.714110:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305914890 00000100:00000001:0.0:1787758187.714112:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305915585 00000100:00000001:0.0:1787758187.714114:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758187.714116:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.714118:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758187.714123:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758187/real 1787758187] req@ffff8ffb45c31500 x1874588659723264/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758187.714131:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758187.714133:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758187.714136:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758187.714137:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758187.714139:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714140:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758187.714142:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758187.714146:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659723264/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758187.714152:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714153:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758187.714155:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714156:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758187.714157:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714158:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758187.714167:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758187.714169:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758187.714171:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758187.714172:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758187.714175:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723264:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758187.714177:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714179:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758187.714180:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758187.714209:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758187.714213:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758187.714214:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758187.714216:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758187.714218:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758187.714219:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758187.714221:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758187.714222:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714225:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758187.714226:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758187.714228:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758187.714229:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758187.714230:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758187.714231:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758187.714233:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758187.714235:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758187.714236:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758187.714238:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714240:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758187.714241:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758187.714243:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758187.714244:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758187.714245:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758187.714247:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758187.714248:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758187.714249:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758187.714251:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758187.714252:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714254:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758187.714256:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758187.714258:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758187.714259:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758187.714260:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758188.289241:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758188.289248:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758188.289249:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758188.289253:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758188.289256:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758188.289259:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758188.289283:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758188.289286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758188.289288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758188.481273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758188.481280:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758188.481286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758188.481288:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758188.481327:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758188.481330:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758188.481332:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758189.313265:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758189.313271:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758189.313272:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758189.313276:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758189.313279:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758189.313283:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758189.313285:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758189.313286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758189.313289:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758189.505302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758189.505307:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758189.505312:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758189.505313:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758189.505329:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758189.505332:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758189.505334:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758190.337389:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758190.337398:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758190.337400:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758190.337407:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758190.337411:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758190.337415:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758190.337419:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758190.337422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758190.337425:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758190.529297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758190.529306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758190.529313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758190.529316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758190.529422:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758190.529428:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758190.529433:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758191.361352:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758191.361361:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758191.361364:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758191.361371:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758191.361376:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758191.361380:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758191.361383:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758191.361387:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758191.361390:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758191.553292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758191.553298:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758191.553303:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758191.553305:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758191.553360:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758191.553364:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758191.553368:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758192.385383:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758192.385394:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758192.385397:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758192.385404:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758192.385409:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758192.385415:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758192.385418:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758192.385422:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758192.385426:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758192.577316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758192.577325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758192.577333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758192.577336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758192.577464:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758192.577470:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758192.577475:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758192.833309:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758192.833323:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758192.833327:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758192.833332:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758192.833338:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758192.833347:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11247 00000100:00080000:3.0:1787758192.833360:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758192.833364:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758192.833367:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758192.833372:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758192.833375:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758192.833380:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758192.833384:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758192.833386:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758192.833388:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758192.833394:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758192.833397:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758192.833401:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758192.833405:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758192.833409:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.833412:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758192.833415:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758192.833428:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758192.833431:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.833435:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758192.833438:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758192.833443:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.833458:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758192.833470:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659723392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758192.833485:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758192.833490:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.833515:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758192.833519:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758192.833524:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758192.833526:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758192.833528:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758192.833532:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758192.833571:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11247 00000100:00080000:3.0:1787758192.833578:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758192.833581:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758192.833584:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758192.833587:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758192.833590:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758192.833593:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758192.833596:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758192.833600:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758192.833601:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758192.833606:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787758192.833607:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758192.833616:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.833617:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787758192.833622:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758192.833624:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758192.833630:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758192.833633:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.833639:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758192.833643:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758192.833645:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758192.833648:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758192.833660:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758192.833666:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659723392:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758192.833670:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758192.833679:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00100000:3.0:1787758192.833683:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:0.0:1787758192.833686:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758192.833694:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787758192.833697:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.833699:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758192.833708:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.833712:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758192.833723:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758192.833730:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.833736:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758192.833738:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787758192.833740:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758192.833744:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 02000000:00000001:3.0:1787758192.833748:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.833752:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758192.833764:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758192.833778:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659723520/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758192.833790:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758192.833794:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.833799:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758192.833804:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758192.833809:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758192.833811:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758192.833813:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758192.833817:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758192.833820:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11247 00000100:00080000:3.0:1787758192.833827:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758192.833831:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.833833:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758192.833838:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758192.833843:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305920705 00000020:00000001:3.0:1787758192.833844:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758192.833850:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758192.833854:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758192.833858:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758192.833861:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758192.833862:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758192.833864:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758192.833875:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758192.833877:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305920705 00000100:00000001:3.0:1787758192.833880:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758192.833887:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758192.833890:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787758192.833893:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758192.833899:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758192.833900:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.833913:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.833914:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758192.833921:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758192.833923:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787758192.833929:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.833931:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758192.833936:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758192.833945:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758192.833946:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787758192.833947:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.833953:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:3.0:1787758192.833955:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758192.833960:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758192.833964:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787758192.833966:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.833972:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758192.833977:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758192.833978:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.833981:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758192.833987:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758192.833994:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659723648/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758192.833995:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758192.834002:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758192.834006:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758192.834013:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723520:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758192.834024:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758192.834029:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758192.834034:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.834037:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758192.834039:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758192.834045:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.834049:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.834053:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758192.834061:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758192.834064:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787758192.834066:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758192.834070:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:3.0:1787758192.834072:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758192.834079:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758192.834083:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758192.834087:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758192.834095:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11247 00000100:00080000:3.0:1787758192.834101:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758192.834105:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758192.834108:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758192.834109:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758192.834112:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758192.834118:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758192.834119:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305920013 00000020:00000001:3.0:1787758192.834122:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758192.834132:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305920705 00000020:00000001:3.0:1787758192.834133:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758192.834145:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305920013 00000100:00000001:3.0:1787758192.834146:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758192.834151:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758192.834154:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305920705 00000100:00080000:3.0:1787758192.834162:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758192.834164:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758192.834170:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834172:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758192.834177:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758192.834246:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834253:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758192.834254:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758192.834258:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787758192.834261:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834264:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:3.0:1787758192.834268:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758192.834272:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787758192.834275:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758192.834280:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:3.0:1787758192.834282:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834289:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758192.834289:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834299:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758192.834303:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758192.834306:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758192.834308:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758192.834320:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758192.834324:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758192.834324:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758192.834333:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.834335:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758192.834342:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.834345:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758192.834355:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659723776/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758192.834362:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758192.834381:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758192.834385:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758192.834387:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.834392:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758192.834397:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.834399:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.834406:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758192.834413:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834427:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758192.834431:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:0.0:1787758192.834439:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758192.834467:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834475:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305920011 00000100:00000001:0.0:1787758192.834480:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305920705 00000100:00000001:0.0:1787758192.834485:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305920011 00000100:00000001:0.0:1787758192.834489:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305920705 00000100:00000001:0.0:1787758192.834493:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758192.834497:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834502:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834506:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758192.834512:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758192.834515:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787758192.834518:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758192.834521:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758192.834523:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758192.834526:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758192.834528:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.834533:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758192.834537:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758192.834542:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758192.834549:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758192.834556:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758192.834559:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758192.834562:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758192.834566:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758192.834572:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834582:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758192.834584:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:0.0:1787758192.834590:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758192.834609:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834615:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305920011 00000100:00000001:0.0:1787758192.834620:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305920705 00000100:00000001:0.0:1787758192.834624:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305920011 00000100:00000001:0.0:1787758192.834628:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305920705 00000100:00000001:0.0:1787758192.834632:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758192.834636:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834641:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834645:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758192.834650:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758192.834653:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:0.0:1787758192.834656:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758192.834659:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758192.834662:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758192.834666:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758193.409291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758193.409299:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758193.409301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758193.409307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758193.409312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758193.409316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758193.409319:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758193.409321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758193.409324:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758193.601322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758193.601328:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758193.601332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758193.601333:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758193.601346:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758193.601348:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758193.601350:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758194.433270:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758194.433280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758194.433283:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758194.433290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758194.433295:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758194.433299:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758194.433302:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758194.433306:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758194.433309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758194.625301:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758194.625311:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758194.625319:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758194.625325:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758194.625328:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758194.625334:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758194.625339:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758194.625342:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758194.625345:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758194.625349:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758194.625480:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758194.625486:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758194.625491:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758195.457315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758195.457324:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758195.457327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758195.457334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758195.457339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758195.457343:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.457347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758195.457350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.457354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758195.649339:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758195.649348:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758195.649355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758195.649357:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758195.649464:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758195.649473:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758195.649480:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787758195.905532:0:2358:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1787758195.905592:0:2358:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787758195.905613:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787758195.905630:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758195.905649:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758195.905653:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758195.905657:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787758195.905672:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758195.905677:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758195.905680:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758195.905686:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758195.905691:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758195.905694:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758195.905696:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758195.905701:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787758195.905729:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1787758195.905740:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758192/real 1787758195] req@ffff8ffb45c31180 x1874588659723520/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758195.905742:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758195.905748:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758195.905750:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758195.905755:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787758195.905767:0:2358:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787758195.905779:0:2358:0:(events.c:34:request_out_callback()) Process entered 00000400:00000001:1.0:1787758195.905782:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787758195.905786:0:2358:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000400:00000001:1.0:1787758195.905787:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758195.905791:0:2358:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758195.905796:0:2358:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758195.905800:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758195.905819:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758195.905823:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.905827:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758195.905830:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758195.905838:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659723520/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758195.905855:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.905859:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758195.905862:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.905865:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758195.905866:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.905869:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758195.905888:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758195.905892:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758195.905897:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758195.905900:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758195.905904:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723520:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758195.905909:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758195.905913:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758192/real 1787758195] req@ffff8ffb45c30000 x1874588659723392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758195.905920:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758195.905922:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758195.905925:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758195.905927:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758195.905929:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.905931:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758195.905933:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758195.905938:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659723392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758195.905947:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.905949:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758195.905952:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.905954:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758195.905955:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.905958:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758195.905960:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758195.905963:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758195.905967:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758195.905970:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758195.905975:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659723392:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758195.905980:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758195.905983:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758195.905985:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758195.905988:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758195.905995:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758195.905998:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758195.906001:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758195.906005:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758195.906008:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758195.906011:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758195.906012:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906015:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758195.906016:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758195.906018:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758195.906020:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758195.906022:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758195.906023:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758195.906026:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758195.906027:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758195.906029:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758195.906030:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906033:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758195.906036:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758195.906038:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758195.906039:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758195.906046:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758192/real 1787758195] req@ffff8ffb45c33480 x1874588659723776/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758195.906057:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758195.906059:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758195.906063:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758195.906066:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758195.906068:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906071:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758195.906073:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758195.906078:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659723776/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758195.906087:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906089:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758195.906092:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906094:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758195.906095:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906097:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758195.906112:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758195.906115:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758195.906120:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758195.906124:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758195.906146:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723776:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758195.906152:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758195.906157:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758192/real 1787758195] req@ffff8ffb45c33800 x1874588659723648/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758195.906168:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758195.906169:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758195.906173:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758195.906176:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758195.906177:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906179:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758195.906180:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758195.906214:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659723648/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758195.906238:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906241:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758195.906247:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906249:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758195.906251:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906253:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758195.906267:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758195.906270:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758195.906275:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758195.906277:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758195.906281:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659723648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758195.906285:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906288:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758195.906289:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758195.906291:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758195.906294:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758195.906296:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758195.906297:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758195.906299:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758195.906300:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758195.906303:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758195.906304:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906306:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758195.906307:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758195.906309:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758195.906311:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758195.906312:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758195.906313:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758195.906315:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758195.906317:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758195.906318:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758195.906320:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906323:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758195.906331:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758195.906334:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758195.906337:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758195.906338:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758196.481288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758196.481296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758196.481299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758196.481305:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758196.481309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758196.481313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758196.481316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758196.481320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758196.481323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758196.673346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758196.673357:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758196.673361:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758196.673364:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758196.673371:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758196.673374:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758196.673377:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758196.673380:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758196.673385:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758196.673387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758196.673548:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758196.673554:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758196.673558:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758197.505282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758197.505291:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758197.505294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758197.505300:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758197.505304:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758197.505309:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.505312:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758197.505316:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.505320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758197.697316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758197.697324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758197.697330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758197.697332:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758197.697365:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758197.697370:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758197.697375:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758197.953431:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758197.953447:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758197.953452:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758197.953458:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758197.953465:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758197.953479:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11252 00000100:00080000:3.0:1787758197.953495:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758197.953500:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758197.953504:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758197.953511:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758197.953515:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758197.953520:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758197.953524:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758197.953528:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758197.953530:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758197.953539:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758197.953545:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758197.953550:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758197.953556:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758197.953562:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953566:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758197.953571:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758197.953586:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758197.953590:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953595:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758197.953600:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758197.953607:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953625:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758197.953642:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659723904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758197.953662:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758197.953669:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953699:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758197.953704:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758197.953724:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758197.953727:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758197.953731:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758197.953736:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758197.953741:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11252 00000100:00080000:3.0:1787758197.953748:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758197.953752:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758197.953755:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758197.953760:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758197.953764:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758197.953768:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758197.953774:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758197.953779:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758197.953783:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758197.953790:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758197.953795:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758197.953799:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758197.953807:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758197.953812:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1787758197.953813:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953822:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758197.953823:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787758197.953830:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758197.953832:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758197.953838:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000008:00000001:3.0:1787758197.953842:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.953846:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953851:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.953858:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758197.953864:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758197.953868:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758197.953872:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953878:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758197.953881:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758197.953889:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758197.953891:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953907:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758197.953908:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659723904:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758197.953923:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:3.0:1787758197.953926:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659724032/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758197.953933:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758197.953939:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.953948:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758197.953959:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758197.953961:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953965:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.953976:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758197.953982:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758197.953989:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758197.953993:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758197.953994:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758197.954002:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00080000:3.0:1787758197.954004:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758197.954018:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758197.954018:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758197.954029:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11252 00000100:00080000:3.0:1787758197.954037:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758197.954042:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758197.954045:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758197.954050:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758197.954053:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758197.954057:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758197.954062:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758197.954067:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758197.954071:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758197.954078:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758197.954082:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758197.954087:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000800:00000100:1.0:1787758197.954120:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758197.954142:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787758197.954145:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758197.954152:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787758197.954155:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954160:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.954163:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758197.954171:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:3.0:1787758197.954174:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758197.954230:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.954235:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758197.954243:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758197.954246:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.954249:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758197.954252:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758197.954256:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.954266:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954276:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758197.954280:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659724160/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758197.954288:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305925826 00000100:00000001:3.0:1787758197.954308:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758197.954313:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758197.954314:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.954326:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954329:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305925826 00000100:00080000:3.0:1787758197.954333:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758197.954343:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787758197.954345:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758197.954355:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758197.954358:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787758197.954360:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758197.954373:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.954375:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758197.954385:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:3.0:1787758197.954387:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11252 00000100:00080000:3.0:1787758197.954400:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758197.954407:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758197/real 1787758197] req@ffff8ffb45c33800 x1874588659723904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758197.954409:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758197.954415:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758197.954426:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758197.954433:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758197.954441:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787758197.954448:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:3.0:1787758197.954449:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787758197.954456:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758197.954468:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758197.954473:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758197.954475:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758197.954478:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758197.954481:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758197.954486:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758197.954489:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758197.954505:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1787758197.954508:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659723904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758197.954519:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758197.954531:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758197.954542:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758197.954543:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758197.954554:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.954558:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758197.954562:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758197.954564:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758197.954573:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.954577:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758197.954580:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.954585:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758197.954589:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758197.954596:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758197.954606:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758197.954611:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758197.954618:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659723904:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758197.954630:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.954635:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758197.954638:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758197.954642:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758197.954650:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758197.954654:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758197.954658:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758197.954665:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758197.954668:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758197.954673:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758197.954676:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.954681:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.954694:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758197.954699:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954704:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758197.954709:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:3.0:1787758197.954715:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954716:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758197.954724:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758197.954726:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954730:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.954739:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758197.954740:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758197.954744:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758197.954748:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.954755:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758197.954758:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758197.954765:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787758197.954769:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724032:192.168.201.118@tcp:38:kworker.0 02000000:00000001:3.0:1787758197.954773:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954785:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758197.954792:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.954793:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758197.954799:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.954807:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:3.0:1787758197.954814:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659724288/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758197.954818:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954839:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758197.954841:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758197.954845:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:3.0:1787758197.954852:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954860:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758197.954866:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758197.954898:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758197.954913:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758197.954920:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758197.954924:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954927:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758197.954934:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954942:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305925134 00000100:00000001:1.0:1787758197.954949:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305925826 00000100:00000001:1.0:1787758197.954954:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305925134 00000100:00000001:1.0:1787758197.954958:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305925826 00000100:00000001:1.0:1787758197.954962:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758197.954966:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.954970:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758197.954980:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758197/real 1787758197] req@ffff8ffb45c33480 x1874588659724032/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758197.954996:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758197.954999:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758197.955005:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758197.955008:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758197.955011:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955015:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758197.955017:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758197.955024:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659724032/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758197.955038:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955043:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758197.955047:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955050:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758197.955052:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955055:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758197.955080:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758197.955085:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758197.955128:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758197.955133:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758197.955139:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724032:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758197.955153:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758197.955155:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758197.955158:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.955163:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758197.955166:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758197.955170:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758197.955176:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724160:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758197.955182:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758197.955226:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758197.955230:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.955234:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758197.955241:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955253:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758197.955256:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787758197.955263:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758197.955292:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758197.955302:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758197.955306:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758197.955309:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955313:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758197.955318:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955324:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305925132 00000100:00000001:1.0:1787758197.955329:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305925826 00000100:00000001:1.0:1787758197.955335:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305925132 00000100:00000001:1.0:1787758197.955339:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305925826 00000100:00000001:1.0:1787758197.955343:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758197.955348:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955352:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758197.955362:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758197/real 1787758197] req@ffff8ffb45c30000 x1874588659724160/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758197.955381:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758197.955384:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758197.955390:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758197.955393:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758197.955396:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955400:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758197.955402:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758197.955410:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659724160/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758197.955423:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955427:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758197.955432:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955436:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758197.955438:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955442:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758197.955464:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758197.955469:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758197.955476:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758197.955481:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758197.955488:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724160:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758197.955496:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955500:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758197.955503:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758197.955506:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758197.955512:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758197.955515:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758197.955518:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758197.955523:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758197.955525:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758197.955529:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758197.955532:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955536:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758197.955539:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758197.955541:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758197.955545:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758197.955548:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758197.955550:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758197.955553:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758197.955556:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758197.955560:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758197.955562:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955566:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955574:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758197.955577:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955582:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758197.955584:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758197.955587:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758197.955590:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758197.955592:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.955597:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758197.955601:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758197.955605:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758197.955611:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758197.955618:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758197.955622:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758197.955625:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758197.955629:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758197.955634:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955646:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758197.955649:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787758197.955656:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758197.955685:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758197.955694:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758197.955699:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758197.955702:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955705:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758197.955711:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955718:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305925132 00000100:00000001:1.0:1787758197.955723:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305925826 00000100:00000001:1.0:1787758197.955727:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305925132 00000100:00000001:1.0:1787758197.955731:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305925826 00000100:00000001:1.0:1787758197.955736:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758197.955739:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955743:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758197.955752:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758197/real 1787758197] req@ffff8ffb45c31180 x1874588659724288/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758197.955771:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758197.955774:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758197.955779:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758197.955782:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758197.955784:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955788:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758197.955791:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758197.955799:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659724288/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758197.955814:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955818:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758197.955823:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955826:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758197.955829:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955833:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758197.955855:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758197.955861:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758197.955869:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758197.955873:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758197.955880:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724288:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758197.955888:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955892:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758197.955895:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758197.955897:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758197.955900:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758197.955902:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758197.955905:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758197.955908:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758197.955910:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758197.955914:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758197.955917:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955921:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758197.955940:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758197.955943:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758197.955947:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758197.955951:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758198.529320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758198.529332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758198.529336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758198.529343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758198.529348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758198.529354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758198.529357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758198.529361:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758198.529365:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758198.721321:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758198.721328:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758198.721334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758198.721335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758198.721373:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758198.721376:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758198.721379:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758199.553256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758199.553262:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758199.553264:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758199.553269:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758199.553272:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758199.553275:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758199.553277:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758199.553280:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758199.553283:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758199.745309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758199.745317:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758199.745323:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758199.745325:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758199.745448:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758199.745455:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758199.745460:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758200.577325:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758200.577337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758200.577340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758200.577349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758200.577355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758200.577361:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758200.577367:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758200.577372:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758200.577376:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758200.769435:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758200.769447:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758200.769454:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758200.769456:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758200.769559:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758200.769568:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758200.769574:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758201.601306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758201.601313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758201.601315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758201.601319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758201.601322:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758201.601325:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758201.601327:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758201.601329:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758201.601331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758201.793311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758201.793319:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758201.793326:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758201.793327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758201.793353:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758201.793356:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758201.793359:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758202.625241:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758202.625247:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758202.625255:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758202.625259:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758202.625262:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758202.625265:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758202.625268:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758202.625270:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758202.625272:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758202.817297:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758202.817304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758202.817310:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758202.817312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758202.817380:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758202.817384:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758202.817388:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758203.073290:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758203.073299:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758203.073301:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758203.073304:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758203.073308:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758203.073314:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11257 00000100:00080000:3.0:1787758203.073324:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758203.073327:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758203.073329:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758203.073332:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758203.073334:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758203.073337:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758203.073339:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758203.073340:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758203.073341:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758203.073345:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758203.073346:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758203.073349:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758203.073351:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758203.073353:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073355:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758203.073357:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758203.073376:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758203.073377:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073380:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758203.073382:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758203.073386:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073396:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758203.073403:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659724416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758203.073411:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758203.073414:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073439:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758203.073441:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758203.073444:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758203.073445:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758203.073446:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758203.073447:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758203.073449:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11257 00000100:00080000:3.0:1787758203.073452:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758203.073454:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758203.073455:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758203.073457:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758203.073457:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758203.073459:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758203.073460:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758203.073462:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758203.073463:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758203.073465:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758203.073466:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758203.073467:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758203.073471:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758203.073472:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073473:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758203.073477:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758203.073479:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073481:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758203.073483:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758203.073483:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073484:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758203.073485:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758203.073487:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073490:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758203.073494:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659724544/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758203.073499:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758203.073501:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073502:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758203.073505:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758203.073507:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758203.073509:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758203.073510:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787758203.073511:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758203.073515:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758203.073515:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758203.073518:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11257 00000100:00000001:1.0:1787758203.073520:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:3.0:1787758203.073522:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758203.073523:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758203.073524:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758203.073526:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073527:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758203.073530:0:178062:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:1.0:1787758203.073531:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:3.0:1787758203.073532:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787758203.073535:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758203.073535:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758203.073538:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758203.073540:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073541:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758203.073543:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787758203.073547:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659724416:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787758203.073548:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758203.073552:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073555:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758203.073555:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758203.073557:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758203.073560:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758203.073560:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073564:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787758203.073565:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787758203.073566:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758203.073568:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000008:00000001:3.0:1787758203.073570:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073572:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758203.073573:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758203.073575:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758203.073577:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073578:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758203.073579:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758203.073580:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073584:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758203.073587:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659724672/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758203.073589:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758203.073592:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787758203.073597:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758203.073598:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758203.073601:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073605:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758203.073606:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758203.073608:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758203.073609:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758203.073610:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758203.073612:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758203.073614:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11257 00000100:00080000:3.0:1787758203.073616:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758203.073617:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758203.073618:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758203.073619:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758203.073619:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758203.073620:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758203.073622:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758203.073623:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758203.073624:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758203.073626:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758203.073627:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758203.073628:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758203.073630:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758203.073632:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073633:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758203.073634:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758203.073636:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073637:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758203.073639:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758203.073640:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073642:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758203.073644:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000800:00000100:1.0:1787758203.073645:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787758203.073646:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758203.073652:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758203.073655:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659724800/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758203.073659:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758203.073662:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758203.073664:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073665:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758203.073667:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073669:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073674:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758203.073680:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073685:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305930945 00000100:00000001:1.0:1787758203.073695:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758203.073701:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305930945 00000100:00000001:1.0:1787758203.073706:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758203.073710:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758203.073713:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073715:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758203.073724:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758203/real 1787758203] req@ffff8ffb45c31500 x1874588659724416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758203.073737:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758203.073739:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758203.073742:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758203.073743:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758203.073745:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073747:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758203.073748:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758203.073753:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659724416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758203.073759:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073761:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758203.073763:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073765:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758203.073765:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073767:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758203.073769:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758203.073771:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758203.073774:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758203.073776:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758203.073780:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659724416:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758203.073787:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758203.073789:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758203.073790:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758203.073792:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758203.073794:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758203.073797:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758203.073800:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724544:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758203.073804:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758203.073806:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758203.073808:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758203.073810:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758203.073814:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073822:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758203.073823:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787758203.073827:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758203.073845:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758203.073852:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758203.073856:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758203.073858:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073860:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758203.073863:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073868:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305930253 00000100:00000001:1.0:1787758203.073871:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305930945 00000100:00000001:1.0:1787758203.073874:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305930253 00000100:00000001:1.0:1787758203.073877:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305930945 00000100:00000001:1.0:1787758203.073880:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758203.073882:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.073884:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758203.073887:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758203/real 1787758203] req@ffff8ffb45c32a00 x1874588659724544/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758203.073894:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758203.073895:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758203.073898:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758203.073899:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758203.073900:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073902:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758203.073903:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758203.073907:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659724544/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758203.073912:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073914:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758203.073916:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073917:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758203.073918:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073919:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758203.073934:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758203.073937:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758203.073939:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758203.073941:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758203.073944:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724544:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758203.073959:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073961:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758203.073963:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758203.073966:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758203.073969:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758203.073971:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758203.073972:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758203.073976:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758203.073978:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758203.073981:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758203.073983:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.073985:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758203.073986:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758203.073988:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758203.073990:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758203.073992:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758203.073993:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758203.073995:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758203.073996:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758203.073998:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758203.073999:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074014:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074022:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758203.074024:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074027:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758203.074029:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758203.074030:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758203.074032:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758203.074033:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758203.074036:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758203.074038:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758203.074040:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758203.074043:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724672:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758203.074046:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758203.074047:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758203.074049:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758203.074050:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758203.074052:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074068:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758203.074070:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787758203.074073:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758203.074084:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758203.074089:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758203.074091:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758203.074093:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074094:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758203.074097:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074100:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305930250 00000100:00000001:1.0:1787758203.074102:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305930945 00000100:00000001:1.0:1787758203.074104:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305930250 00000100:00000001:1.0:1787758203.074106:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305930945 00000100:00000001:1.0:1787758203.074108:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758203.074110:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074112:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758203.074115:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758203/real 1787758203] req@ffff8ffb45c30700 x1874588659724672/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758203.074121:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758203.074122:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758203.074124:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758203.074125:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758203.074126:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074127:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758203.074128:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758203.074130:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659724672/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758203.074135:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074137:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758203.074138:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074139:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758203.074140:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074141:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758203.074150:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758203.074151:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758203.074153:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758203.074154:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758203.074156:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724672:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758203.074158:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758203.074159:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758203.074160:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758203.074161:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758203.074163:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758203.074164:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758203.074166:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758203.074168:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758203.074169:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758203.074170:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758203.074171:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758203.074173:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074176:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758203.074177:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787758203.074179:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758203.074214:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758203.074218:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758203.074220:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758203.074221:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074222:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758203.074223:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074226:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305930251 00000100:00000001:1.0:1787758203.074227:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305930945 00000100:00000001:1.0:1787758203.074229:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305930251 00000100:00000001:1.0:1787758203.074230:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305930945 00000100:00000001:1.0:1787758203.074232:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758203.074233:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074234:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758203.074237:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758203/real 1787758203] req@ffff8ffb45c30a80 x1874588659724800/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758203.074242:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758203.074243:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758203.074247:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758203.074248:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758203.074249:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074250:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758203.074251:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758203.074253:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659724800/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758203.074262:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074264:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758203.074265:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074266:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758203.074267:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074268:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758203.074281:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758203.074284:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758203.074286:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758203.074288:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758203.074290:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659724800:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758203.074293:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074294:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758203.074296:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758203.074297:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758203.074299:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758203.074300:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758203.074302:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758203.074304:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758203.074305:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758203.074307:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758203.074309:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074310:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758203.074311:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758203.074312:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758203.074313:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758203.074314:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758203.074315:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758203.074316:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758203.074317:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758203.074318:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758203.074319:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074321:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.074324:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758203.074337:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.074339:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758203.074340:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758203.649306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758203.649314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758203.649317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758203.649323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758203.649327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758203.649332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.649336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758203.649339:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758203.649342:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758203.841310:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758203.841317:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758203.841326:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758203.841328:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758203.841386:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758203.841391:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758203.841398:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758204.673311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758204.673319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758204.673322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758204.673328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758204.673331:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758204.673336:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758204.673338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758204.673341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758204.673361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758204.865372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758204.865381:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758204.865389:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758204.865394:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758204.865397:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758204.865401:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758204.865405:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758204.865408:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758204.865411:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758204.865414:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758204.865529:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758204.865535:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758204.865541:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758205.697362:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758205.697372:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758205.697374:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758205.697381:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758205.697385:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758205.697390:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758205.697393:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758205.697396:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758205.697400:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758205.889341:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758205.889352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758205.889362:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758205.889365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758205.889423:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758205.889428:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758205.889434:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758206.721315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758206.721325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758206.721327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758206.721334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758206.721338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758206.721342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758206.721345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758206.721348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758206.721352:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758206.913415:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758206.913424:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758206.913429:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758206.913432:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758206.913438:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758206.913441:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758206.913443:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758206.913446:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758206.913450:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758206.913452:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758206.913532:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758206.913537:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758206.913541:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758207.745270:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758207.745278:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758207.745280:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758207.745286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758207.745289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758207.745294:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758207.745297:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758207.745299:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758207.745302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758207.937302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758207.937308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758207.937315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758207.937316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758207.937374:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758207.937378:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758207.937381:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758208.193255:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758208.193264:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758208.193267:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758208.193270:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758208.193292:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758208.193300:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11262 00000100:00080000:3.0:1787758208.193313:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758208.193317:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758208.193320:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758208.193325:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758208.193329:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758208.193333:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758208.193336:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758208.193339:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758208.193341:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758208.193347:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758208.193350:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758208.193353:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758208.193358:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758208.193364:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193383:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758208.193387:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758208.193400:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758208.193402:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193407:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758208.193410:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758208.193415:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193429:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758208.193457:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659724928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758208.193470:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758208.193474:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193490:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758208.193493:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758208.193496:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758208.193497:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758208.193499:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758208.193501:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758208.193504:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11262 00000100:00080000:3.0:1787758208.193508:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758208.193509:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758208.193511:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758208.193513:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758208.193514:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758208.193516:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758208.193517:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758208.193519:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758208.193520:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758208.193523:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758208.193525:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758208.193527:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758208.193531:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758208.193534:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193537:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758208.193542:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758208.193546:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193548:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758208.193553:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758208.193554:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193556:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758208.193557:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758208.193559:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193563:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758208.193570:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659725056/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758208.193578:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758208.193580:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193582:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.193583:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:3.0:1787758208.193585:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758208.193593:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758208.193594:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758208.193596:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787758208.193597:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758208.193602:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758208.193603:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758208.193617:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787758208.193618:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11262 00000100:00000001:1.0:1787758208.193623:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758208.193626:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787758208.193635:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758208.193652:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758208.193654:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758208.193658:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193662:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787758208.193676:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659724928:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787758208.193678:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758208.193688:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758208.193692:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:3.0:1787758208.193697:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758208.193701:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758208.193709:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758208.193711:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758208.193715:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758208.193726:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758208.193748:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193748:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758208.193767:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758208.193771:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758208.193773:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758208.193778:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758208.193781:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193783:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758208.193787:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758208.193790:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.193791:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758208.193810:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.193812:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 02000000:00000001:3.0:1787758208.193826:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758208.193828:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787758208.193830:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193834:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758208.193836:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758208.193838:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193845:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758208.193850:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659725184/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758208.193858:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758208.193861:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.193865:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758208.193868:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758208.193871:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758208.193873:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758208.193874:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758208.193878:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758208.193881:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11262 00000100:00080000:3.0:1787758208.193888:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758208.193890:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758208.193892:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758208.193894:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758208.193895:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758208.193897:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758208.193899:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000800:00000100:1.0:1787758208.193906:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758208.193907:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758208.193946:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758208.193957:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758208.193961:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758208.193964:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758208.193969:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787758208.193970:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758208.193978:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.193983:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758208.193983:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758208.193988:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787758208.193991:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758208.193994:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758208.194006:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787758208.194011:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194014:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305936065 00000100:00000001:3.0:1787758208.194016:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758208.194032:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758208.194035:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758208.194037:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758208.194039:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758208.194041:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194043:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758208.194048:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758208.194057:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659725312/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758208.194059:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305936065 00000100:00000001:1.0:1787758208.194090:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758208.194103:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758208.194107:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194109:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758208.194112:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194122:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194129:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758208.194141:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758208/real 1787758208] req@ffff8ffb45c33b80 x1874588659724928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758208.194158:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758208.194160:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758208.194166:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758208.194170:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758208.194172:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194175:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758208.194177:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758208.194264:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659724928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758208.194275:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194279:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758208.194283:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194285:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758208.194286:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194289:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758208.194292:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758208.194296:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758208.194302:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758208.194304:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758208.194310:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659724928:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758208.194316:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758208.194319:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758208.194321:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758208.194324:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758208.194327:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758208.194330:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758208.194334:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725056:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758208.194339:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758208.194342:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758208.194344:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758208.194346:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758208.194351:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194359:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758208.194361:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787758208.194368:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758208.194388:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758208.194394:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758208.194397:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758208.194399:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194401:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758208.194405:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194411:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305935374 00000100:00000001:1.0:1787758208.194414:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305936065 00000100:00000001:1.0:1787758208.194417:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305935374 00000100:00000001:1.0:1787758208.194420:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305936065 00000100:00000001:1.0:1787758208.194423:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758208.194425:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194427:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758208.194432:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758208/real 1787758208] req@ffff8ffb45c30e00 x1874588659725056/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758208.194442:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758208.194444:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758208.194446:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758208.194448:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758208.194450:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194452:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758208.194454:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758208.194458:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659725056/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758208.194468:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194471:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758208.194474:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194475:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758208.194477:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194479:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758208.194498:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758208.194501:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758208.194505:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758208.194507:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758208.194511:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725056:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758208.194516:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194519:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758208.194521:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758208.194525:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758208.194529:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758208.194532:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758208.194534:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758208.194540:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758208.194543:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758208.194546:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758208.194548:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194550:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758208.194551:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758208.194553:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758208.194556:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758208.194558:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758208.194559:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758208.194561:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758208.194562:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758208.194564:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758208.194565:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194568:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194577:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758208.194580:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194583:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758208.194585:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758208.194587:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758208.194589:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758208.194591:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758208.194594:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758208.194596:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758208.194598:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758208.194602:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725184:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758208.194606:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758208.194609:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758208.194611:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758208.194613:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758208.194616:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194632:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758208.194634:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:1.0:1787758208.194639:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758208.194653:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758208.194658:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758208.194660:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758208.194662:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194664:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758208.194667:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194671:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305935371 00000100:00000001:1.0:1787758208.194674:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305936065 00000100:00000001:1.0:1787758208.194677:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305935371 00000100:00000001:1.0:1787758208.194680:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305936065 00000100:00000001:1.0:1787758208.194683:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758208.194685:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194687:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758208.194691:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758208/real 1787758208] req@ffff8ffb45c31c00 x1874588659725184/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758208.194708:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758208.194710:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758208.194713:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758208.194715:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758208.194716:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194718:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758208.194719:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758208.194723:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659725184/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758208.194743:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194747:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758208.194752:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194755:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758208.194758:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.194761:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758208.194784:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758208.194789:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758208.194807:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758208.194812:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758208.194819:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725184:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758208.194827:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758208.194830:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758208.194833:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758208.194839:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758208.194841:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758208.194843:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758208.194848:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758208.194852:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758208.194855:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758208.194858:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758208.194870:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758208.194874:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194881:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758208.194883:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787758208.194889:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758208.194904:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758208.194910:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758208.194948:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758208.194952:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194955:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758208.194961:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194967:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305935371 00000100:00000001:1.0:1787758208.194972:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305936065 00000100:00000001:1.0:1787758208.194976:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305935371 00000100:00000001:1.0:1787758208.194981:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305936065 00000100:00000001:1.0:1787758208.194985:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758208.194989:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.194991:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758208.194998:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758208/real 1787758208] req@ffff8ffb45c30a80 x1874588659725312/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758208.195008:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758208.195010:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758208.195014:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758208.195016:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758208.195018:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.195022:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758208.195025:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758208.195031:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659725312/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758208.195044:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.195048:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758208.195054:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.195057:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758208.195059:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.195063:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758208.195099:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758208.195105:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758208.195112:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758208.195116:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758208.195121:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725312:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758208.195125:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.195128:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758208.195130:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758208.195132:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758208.195135:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758208.195137:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758208.195139:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758208.195142:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758208.195144:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758208.195146:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758208.195148:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.195150:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758208.195151:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758208.195153:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758208.195155:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758208.195156:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758208.195157:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758208.195159:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758208.195160:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758208.195162:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758208.195163:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.195165:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.195169:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758208.195171:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.195173:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758208.195175:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758208.769327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758208.769338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758208.769341:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758208.769349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758208.769354:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758208.769358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.769362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758208.769365:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758208.769368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758208.961260:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758208.961266:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758208.961280:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758208.961285:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758208.961332:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758208.961335:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758208.961338:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758209.793372:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758209.793383:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758209.793386:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758209.793394:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758209.793399:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758209.793406:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758209.793410:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758209.793415:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758209.793420:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758209.985272:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758209.985278:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758209.985283:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758209.985284:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758209.985301:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758209.985303:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758209.985305:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758210.817323:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758210.817332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758210.817335:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758210.817342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758210.817347:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758210.817352:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758210.817355:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758210.817358:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758210.817362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758211.009307:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758211.009319:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758211.009332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758211.009336:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758211.009406:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758211.009412:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758211.009417:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758211.841263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758211.841269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758211.841271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758211.841275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758211.841278:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758211.841281:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758211.841283:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758211.841286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758211.841288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758212.033248:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758212.033253:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758212.033258:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758212.033260:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758212.033297:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758212.033300:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758212.033303:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758212.865307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758212.865316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758212.865319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758212.865326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758212.865331:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758212.865337:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758212.865341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758212.865344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758212.865347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758213.057377:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758213.057387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758213.057396:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758213.057399:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758213.057460:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758213.057466:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758213.057471:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758213.313320:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758213.313328:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758213.313330:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758213.313334:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758213.313337:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758213.313343:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11268 00000100:00080000:2.0:1787758213.313353:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758213.313355:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758213.313357:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758213.313360:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758213.313362:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758213.313364:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758213.313366:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758213.313367:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758213.313368:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758213.313372:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758213.313373:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758213.313376:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758213.313378:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758213.313380:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313382:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758213.313383:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758213.313393:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758213.313394:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313397:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758213.313399:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758213.313402:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313412:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758213.313419:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659725440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758213.313427:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758213.313430:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313444:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758213.313446:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758213.313450:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758213.313451:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758213.313453:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758213.313455:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758213.313457:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11268 00000100:00080000:2.0:1787758213.313459:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758213.313460:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758213.313461:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758213.313462:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758213.313463:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758213.313465:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758213.313467:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758213.313468:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758213.313470:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758213.313472:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758213.313474:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758213.313475:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758213.313478:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758213.313480:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313481:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758213.313484:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:2.0:1787758213.313486:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758213.313490:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:2.0:1787758213.313490:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313493:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313494:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787758213.313497:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758213.313498:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787758213.313500:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313501:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313505:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758213.313507:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758213.313508:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758213.313511:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758213.313512:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758213.313518:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313520:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758213.313524:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659725440:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1787758213.313528:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659725568/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758213.313530:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758213.313535:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758213.313538:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758213.313542:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758213.313547:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758213.313549:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313549:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313556:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758213.313558:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758213.313560:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758213.313561:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758213.313562:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758213.313564:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758213.313566:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11268 00000100:00000001:1.0:1787758213.313567:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787758213.313568:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758213.313570:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:2.0:1787758213.313571:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758213.313575:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758213.313576:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787758213.313577:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758213.313580:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758213.313581:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758213.313582:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758213.313584:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758213.313585:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758213.313587:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758213.313589:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758213.313591:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758213.313592:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758213.313594:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313595:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758213.313597:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758213.313599:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313600:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758213.313603:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758213.313604:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313606:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758213.313607:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758213.313608:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758213.313611:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758213.313614:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758213.313620:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659725696/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758213.313622:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758213.313627:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758213.313628:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758213.313630:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313631:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313635:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758213.313636:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758213.313639:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758213.313641:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758213.313641:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758213.313645:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758213.313646:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305941185 00000100:00080000:2.0:1787758213.313647:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758213.313652:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758213.313654:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11268 00000100:00080000:2.0:1787758213.313657:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758213.313658:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758213.313658:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758213.313662:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758213.313665:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758213.313666:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305941185 00000020:00000001:2.0:1787758213.313666:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758213.313669:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758213.313672:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:2.0:1787758213.313673:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758213.313676:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758213.313677:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758213.313679:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758213.313681:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758213.313682:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758213.313683:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758213.313684:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758213.313690:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758213.313690:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313696:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758213.313696:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758213.313700:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758213.313705:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758213/real 1787758213] req@ffff8ffb47a38000 x1874588659725440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:2.0:1787758213.313705:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313707:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758213.313713:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758213.313715:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313718:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000400:00000001:1.0:1787758213.313719:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:2.0:1787758213.313720:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000400:00000001:1.0:1787758213.313724:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:2.0:1787758213.313724:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313729:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758213.313730:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313733:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758213.313736:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758213.313737:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659725824/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758213.313739:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758213.313742:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758213.313749:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:1.0:1787758213.313750:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659725440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758213.313753:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758213.313758:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313764:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313768:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758213.313770:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313771:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758213.313772:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313773:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758213.313775:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758213.313777:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758213.313781:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758213.313783:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758213.313786:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659725440:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758213.313790:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313792:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758213.313793:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758213.313794:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758213.313797:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758213.313798:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758213.313800:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758213.313803:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758213.313804:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758213.313806:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758213.313807:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313809:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313815:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758213.313816:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313817:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758213.313818:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758213.313820:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758213.313821:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758213.313837:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758213.313840:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758213.313841:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758213.313842:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758213.313845:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725568:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758213.313847:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758213.313849:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758213.313850:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758213.313851:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758213.313853:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313857:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758213.313858:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:1.0:1787758213.313862:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758213.313871:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758213.313874:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758213.313876:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758213.313877:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313878:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758213.313880:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313883:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305940493 00000100:00000001:1.0:1787758213.313885:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305941185 00000100:00000001:1.0:1787758213.313887:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305940493 00000100:00000001:1.0:1787758213.313888:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305941185 00000100:00000001:1.0:1787758213.313889:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758213.313891:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313892:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758213.313895:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758213/real 1787758213] req@ffff8ffb47a38380 x1874588659725568/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758213.313901:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758213.313902:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758213.313903:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758213.313904:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758213.313905:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313906:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758213.313908:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758213.313910:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659725568/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758213.313914:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313916:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758213.313917:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313918:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758213.313919:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313920:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758213.313931:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758213.313932:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758213.313934:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758213.313935:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758213.313937:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725568:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758213.313939:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758213.313940:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758213.313941:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758213.313942:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758213.313944:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758213.313945:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758213.313947:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758213.313948:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758213.313949:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758213.313951:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758213.313952:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758213.313954:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313957:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758213.313958:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:1.0:1787758213.313960:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758213.313966:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758213.313969:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758213.313970:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758213.313971:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313972:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758213.313973:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313975:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305940490 00000100:00000001:1.0:1787758213.313977:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305941185 00000100:00000001:1.0:1787758213.313978:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305940490 00000100:00000001:1.0:1787758213.313980:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305941185 00000100:00000001:1.0:1787758213.313982:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758213.313983:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.313984:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758213.313986:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758213/real 1787758213] req@ffff8ffb47a39880 x1874588659725696/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758213.313990:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758213.313991:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758213.313992:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758213.313993:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758213.313994:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.313995:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758213.313996:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758213.313998:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659725696/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758213.314003:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314004:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758213.314006:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314007:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758213.314008:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314009:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758213.314018:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758213.314020:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758213.314022:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758213.314023:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758213.314025:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758213.314027:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758213.314028:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758213.314028:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758213.314029:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758213.314030:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758213.314032:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758213.314033:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725824:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758213.314035:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758213.314036:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758213.314037:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758213.314038:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758213.314039:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.314042:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758213.314043:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787758213.314045:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758213.314050:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758213.314053:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758213.314054:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758213.314055:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.314056:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758213.314058:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.314059:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305940490 00000100:00000001:1.0:1787758213.314061:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305941185 00000100:00000001:1.0:1787758213.314063:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305940490 00000100:00000001:1.0:1787758213.314065:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305941185 00000100:00000001:1.0:1787758213.314066:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758213.314067:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.314068:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758213.314070:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758213/real 1787758213] req@ffff8ffb47a3a680 x1874588659725824/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758213.314075:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758213.314076:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758213.314077:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758213.314078:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758213.314078:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314079:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758213.314080:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758213.314082:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659725824/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758213.314086:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314087:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758213.314089:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314090:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758213.314091:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314092:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758213.314100:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758213.314102:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758213.314104:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758213.314105:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758213.314108:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659725824:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758213.314110:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314111:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758213.314112:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758213.314113:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758213.314115:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758213.314116:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758213.314117:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758213.314119:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758213.314120:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758213.314123:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758213.314124:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314126:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758213.314127:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758213.314128:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758213.314131:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758213.314132:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758213.314133:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758213.314135:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758213.314136:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758213.314138:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758213.314140:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314142:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758213.314143:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758213.314144:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758213.314147:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758213.314148:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758213.314149:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758213.314150:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758213.314151:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758213.314154:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758213.314155:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314157:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.314161:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758213.314162:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.314164:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758213.314166:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758213.889252:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758213.889259:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758213.889262:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758213.889267:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758213.889271:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758213.889275:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.889278:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758213.889281:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758213.889285:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758214.081288:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758214.081296:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758214.081304:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758214.081306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758214.081333:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758214.081337:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758214.081340:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758214.913264:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758214.913270:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758214.913271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758214.913275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758214.913278:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758214.913281:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758214.913284:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758214.913286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758214.913288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758215.105277:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758215.105282:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758215.105287:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758215.105290:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758215.105292:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758215.105294:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758215.105296:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758215.105298:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758215.105300:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758215.105301:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758215.105349:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758215.105352:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758215.105355:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758215.937251:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758215.937258:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758215.937260:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758215.937264:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758215.937267:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758215.937269:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758215.937271:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758215.937274:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758215.937276:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758216.129260:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758216.129266:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758216.129270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758216.129271:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758216.129315:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758216.129319:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758216.129321:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758216.961308:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758216.961317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758216.961319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758216.961325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758216.961329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758216.961333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758216.961336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758216.961338:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758216.961341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758217.153273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758217.153280:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758217.153283:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758217.153285:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758217.153290:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758217.153292:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758217.153294:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758217.153296:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758217.153299:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758217.153300:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758217.153389:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758217.153392:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758217.153395:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758217.985257:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758217.985263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758217.985265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758217.985269:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758217.985272:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758217.985275:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758217.985277:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758217.985279:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758217.985281:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758218.177274:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758218.177278:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758218.177282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758218.177283:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758218.177299:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758218.177301:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758218.177303:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758218.433306:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758218.433315:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758218.433317:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758218.433320:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758218.433324:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758218.433330:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11273 00000100:00080000:1.0:1787758218.433339:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758218.433342:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758218.433343:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758218.433347:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758218.433349:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758218.433351:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758218.433353:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758218.433355:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758218.433356:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758218.433359:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758218.433361:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758218.433362:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758218.433365:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758218.433367:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433369:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758218.433371:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758218.433380:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758218.433382:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433384:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758218.433386:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758218.433390:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433401:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758218.433412:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659725952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758218.433423:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758218.433428:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433448:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758218.433452:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758218.433456:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758218.433458:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758218.433459:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758218.433463:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758218.433465:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11273 00000100:00080000:1.0:1787758218.433468:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758218.433469:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758218.433470:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758218.433472:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758218.433473:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758218.433475:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758218.433476:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758218.433478:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758218.433479:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758218.433481:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758218.433483:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758218.433484:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758218.433489:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758218.433490:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433492:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758218.433496:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758218.433498:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433500:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433503:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787758218.433504:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758218.433507:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433508:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758218.433509:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758218.433512:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758218.433512:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758218.433514:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433516:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758218.433518:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433520:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.433522:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758218.433526:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787758218.433526:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659726080/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758218.433530:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758218.433538:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659725952:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758218.433546:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787758218.433549:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758218.433550:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758218.433552:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.433553:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.433556:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758218.433557:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758218.433560:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758218.433561:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758218.433563:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758218.433565:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758218.433567:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758218.433569:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758218.433572:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11273 00000100:00080000:1.0:1787758218.433575:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758218.433576:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758218.433578:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758218.433578:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758218.433581:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758218.433582:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000020:00000001:1.0:1787758218.433585:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758218.433588:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:1.0:1787758218.433588:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758218.433592:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758218.433593:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758218.433595:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758218.433597:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758218.433598:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758218.433599:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758218.433601:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758218.433603:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433604:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758218.433606:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758218.433608:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433609:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758218.433613:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758218.433614:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433616:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758218.433617:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758218.433618:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433623:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758218.433624:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1787758218.433632:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659726208/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758218.433637:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758218.433642:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758218.433644:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433644:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758218.433648:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758218.433650:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433654:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433655:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433658:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305946305 00000100:00080000:1.0:1787758218.433660:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758218.433665:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758218.433668:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758218.433668:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758218.433672:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758218.433676:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305946305 00000100:00000001:1.0:1787758218.433678:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758218.433681:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787758218.433682:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11273 00000100:00080000:1.0:1787758218.433687:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758218.433689:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758218.433690:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758218.433691:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787758218.433692:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758218.433695:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758218.433696:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758218.433697:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758218.433700:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:1.0:1787758218.433701:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758218.433704:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758218.433706:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:0.0:1787758218.433707:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758218/real 1787758218] req@ffff8ffb4a536300 x1874588659725952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758218.433711:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758218.433717:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758218.433721:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000400:00000001:0.0:1787758218.433724:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:1.0:1787758218.433725:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:0.0:1787758218.433728:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:1.0:1787758218.433729:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758218.433749:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758218.433750:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758218.433752:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000008:00000001:1.0:1787758218.433753:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433757:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758218.433757:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433760:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758218.433761:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758218.433764:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:1.0:1787758218.433765:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433767:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758218.433770:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:0.0:1787758218.433775:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659725952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787758218.433775:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758218.433780:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433786:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433788:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758218.433790:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433791:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433795:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758218.433796:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433798:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758218.433801:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1787758218.433802:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659726336/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758218.433804:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758218.433810:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758218.433814:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758218.433821:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659725952:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758218.433821:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758218.433825:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433826:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758218.433828:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433830:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758218.433832:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758218.433834:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758218.433837:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758218.433839:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758218.433840:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758218.433845:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758218.433846:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758218.433848:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758218.433850:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433851:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433857:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758218.433858:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433860:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758218.433861:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758218.433862:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758218.433863:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758218.433864:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.433866:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758218.433868:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758218.433869:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758218.433871:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726080:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758218.433874:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758218.433875:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758218.433876:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.433877:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758218.433879:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433884:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758218.433885:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:0.0:1787758218.433888:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758218.433898:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758218.433901:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758218.433903:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758218.433904:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433905:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758218.433907:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433909:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305945613 00000100:00000001:0.0:1787758218.433911:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305946305 00000100:00000001:0.0:1787758218.433913:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305945613 00000100:00000001:0.0:1787758218.433915:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305946305 00000100:00000001:0.0:1787758218.433916:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758218.433917:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433918:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758218.433921:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758218/real 1787758218] req@ffff8ffb4a534a80 x1874588659726080/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758218.433926:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758218.433927:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758218.433928:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758218.433929:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758218.433930:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433931:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758218.433932:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758218.433935:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659726080/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758218.433940:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433941:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758218.433943:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433944:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758218.433945:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.433946:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758218.433955:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758218.433956:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758218.433958:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758218.433960:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758218.433961:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726080:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758218.433964:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758218.433965:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758218.433966:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.433967:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758218.433969:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758218.433970:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758218.433972:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758218.433973:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758218.433974:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758218.433975:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.433976:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758218.433978:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433981:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758218.433982:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758218.433985:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758218.433990:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758218.433993:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758218.433994:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758218.433996:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.433997:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758218.433998:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.434001:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305945610 00000100:00000001:0.0:1787758218.434002:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305946305 00000100:00000001:0.0:1787758218.434004:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305945610 00000100:00000001:0.0:1787758218.434006:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305946305 00000100:00000001:0.0:1787758218.434007:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758218.434008:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.434009:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758218.434011:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758218/real 1787758218] req@ffff8ffb4a537b80 x1874588659726208/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758218.434016:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758218.434017:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758218.434018:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758218.434019:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758218.434020:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434021:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758218.434021:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758218.434024:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659726208/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758218.434028:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434029:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758218.434030:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434032:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758218.434032:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434033:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758218.434042:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758218.434043:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758218.434045:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758218.434046:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758218.434048:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758218.434051:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758218.434052:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758218.434053:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.434054:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758218.434055:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758218.434056:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758218.434058:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758218.434059:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758218.434060:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758218.434061:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758218.434062:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758218.434064:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.434066:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758218.434067:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758218.434069:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758218.434075:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758218.434077:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758218.434078:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758218.434079:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.434080:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758218.434082:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.434083:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305945610 00000100:00000001:0.0:1787758218.434085:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305946305 00000100:00000001:0.0:1787758218.434087:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305945610 00000100:00000001:0.0:1787758218.434088:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305946305 00000100:00000001:0.0:1787758218.434090:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758218.434091:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.434092:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758218.434094:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758218/real 1787758218] req@ffff8ffb4a537800 x1874588659726336/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758218.434103:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758218.434104:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758218.434105:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758218.434106:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758218.434107:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434108:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758218.434109:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758218.434111:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659726336/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758218.434115:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434116:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758218.434117:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434118:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758218.434119:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434121:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758218.434129:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758218.434131:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758218.434133:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758218.434135:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758218.434137:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726336:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758218.434139:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434140:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758218.434140:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758218.434141:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758218.434143:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758218.434143:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758218.434144:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758218.434147:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758218.434147:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758218.434149:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758218.434150:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434151:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758218.434152:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758218.434153:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758218.434155:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758218.434156:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758218.434156:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758218.434158:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758218.434158:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758218.434159:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758218.434160:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434161:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758218.434162:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758218.434162:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758218.434164:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758218.434164:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758218.434165:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758218.434166:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758218.434167:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758218.434167:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758218.434168:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434169:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758218.434171:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758218.434172:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758218.434173:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758218.434174:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758219.009264:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758219.009270:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758219.009272:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758219.009276:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758219.009279:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758219.009282:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758219.009284:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758219.009286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758219.009288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758219.201282:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758219.201290:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758219.201296:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758219.201297:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758219.201334:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758219.201337:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758219.201340:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758220.033246:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758220.033254:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758220.033256:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758220.033261:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758220.033265:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758220.033269:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758220.033272:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758220.033274:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758220.033277:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758220.225269:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758220.225275:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758220.225279:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758220.225280:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758220.225325:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758220.225329:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758220.225331:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758221.057311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758221.057337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758221.057340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758221.057346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758221.057350:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758221.057355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758221.057357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758221.057360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758221.057364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758221.249273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758221.249279:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758221.249284:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758221.249286:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758221.249356:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758221.249361:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758221.249365:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758222.081303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758222.081312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758222.081315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758222.081322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758222.081326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758222.081331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758222.081334:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758222.081337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758222.081341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758222.273336:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758222.273347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758222.273358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758222.273360:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758222.273389:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758222.273392:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758222.273394:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758223.105267:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758223.105274:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758223.105277:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758223.105283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758223.105287:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758223.105291:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758223.105294:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758223.105296:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758223.105299:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758223.297265:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758223.297270:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758223.297275:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758223.297276:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758223.297325:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758223.297328:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758223.297331:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758223.553263:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758223.553271:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758223.553273:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758223.553276:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758223.553280:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758223.553286:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11278 00000100:00080000:1.0:1787758223.553296:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758223.553299:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758223.553301:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758223.553304:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758223.553306:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758223.553309:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758223.553311:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758223.553313:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758223.553314:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758223.553317:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758223.553320:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758223.553322:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758223.553325:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758223.553327:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553329:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758223.553330:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758223.553340:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758223.553342:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553344:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758223.553346:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758223.553350:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553360:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758223.553367:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659726464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758223.553376:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758223.553379:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553393:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758223.553395:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758223.553398:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758223.553399:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758223.553400:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758223.553403:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758223.553404:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11278 00000100:00080000:1.0:1787758223.553406:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758223.553407:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758223.553409:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758223.553411:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758223.553412:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758223.553414:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758223.553416:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758223.553418:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758223.553420:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758223.553422:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758223.553424:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758223.553425:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758223.553429:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758223.553431:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787758223.553431:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553433:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758223.553439:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:1.0:1787758223.553439:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758223.553443:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553444:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758223.553445:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553451:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787758223.553451:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758223.553454:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553456:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553456:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758223.553459:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758223.553461:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553465:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758223.553467:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659726592/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758223.553472:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758223.553474:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758223.553476:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758223.553477:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758223.553481:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553483:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758223.553486:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758223.553489:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659726464:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787758223.553490:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758223.553494:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758223.553497:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758223.553498:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787758223.553502:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787758223.553504:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:1.0:1787758223.553506:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11278 02000000:00000001:0.0:1787758223.553507:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758223.553510:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758223.553511:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758223.553515:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:0.0:1787758223.553517:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553517:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758223.553519:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758223.553521:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758223.553523:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758223.553524:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758223.553526:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758223.553527:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758223.553529:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758223.553530:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758223.553531:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758223.553533:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787758223.553533:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758223.553536:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 02000000:00000001:1.0:1787758223.553537:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553542:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758223.553544:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00100000:1.0:1787758223.553545:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758223.553548:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553549:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758223.553553:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758223.553554:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553555:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758223.553556:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758223.553557:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553560:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758223.553563:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659726720/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758223.553569:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758223.553571:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553573:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758223.553575:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758223.553577:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758223.553578:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758223.553579:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758223.553581:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000800:00000100:0.0:1787758223.553586:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787758223.553586:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11278 00000100:00080000:1.0:1787758223.553592:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758223.553594:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758223.553595:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758223.553597:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758223.553598:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758223.553599:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758223.553601:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:1.0:1787758223.553601:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758223.553605:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758223.553606:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758223.553607:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787758223.553610:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758223.553611:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553614:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758223.553614:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758223.553618:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758223.553621:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758223.553622:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758223.553626:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553627:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305951425 00000100:00000001:1.0:1787758223.553628:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758223.553630:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758223.553632:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553633:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758223.553635:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758223.553637:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553638:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758223.553638:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758223.553639:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553658:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758223.553662:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659726848/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758223.553665:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758223.553671:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758223.553673:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305951425 00000100:00000001:1.0:1787758223.553673:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758223.553675:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553680:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758223.553690:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758223.553693:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553696:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758223.553701:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758223/real 1787758223] req@ffff8ffb4a535c00 x1874588659726464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758223.553710:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758223.553712:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758223.553715:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758223.553718:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758223.553719:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553721:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758223.553722:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758223.553726:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659726464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758223.553732:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553735:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758223.553737:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553738:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758223.553738:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553740:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758223.553742:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758223.553744:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758223.553746:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758223.553748:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758223.553751:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659726464:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758223.553754:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553756:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758223.553757:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758223.553759:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758223.553763:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758223.553764:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758223.553766:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758223.553770:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758223.553771:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758223.553772:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758223.553774:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553775:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553781:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758223.553782:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553784:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758223.553785:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758223.553786:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758223.553787:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758223.553788:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758223.553789:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758223.553791:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758223.553792:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758223.553795:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726592:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758223.553797:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758223.553798:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758223.553799:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758223.553801:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758223.553802:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553806:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758223.553807:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758223.553811:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758223.553820:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758223.553823:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758223.553825:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758223.553826:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553827:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758223.553829:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553831:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305950733 00000100:00000001:0.0:1787758223.553833:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305951425 00000100:00000001:0.0:1787758223.553835:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305950733 00000100:00000001:0.0:1787758223.553837:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305951425 00000100:00000001:0.0:1787758223.553838:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758223.553839:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553840:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758223.553843:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758223/real 1787758223] req@ffff8ffb4a534700 x1874588659726592/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758223.553850:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758223.553850:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758223.553852:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758223.553853:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758223.553854:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553855:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758223.553856:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758223.553859:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659726592/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758223.553863:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553864:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758223.553866:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553867:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758223.553868:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553869:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758223.553878:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758223.553880:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758223.553882:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758223.553883:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758223.553885:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726592:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758223.553887:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758223.553888:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758223.553889:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758223.553890:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758223.553891:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758223.553893:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758223.553895:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758223.553897:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758223.553898:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758223.553899:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758223.553900:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758223.553902:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553905:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758223.553906:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758223.553909:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758223.553915:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758223.553917:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758223.553919:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758223.553920:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553921:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758223.553922:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553924:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305950730 00000100:00000001:0.0:1787758223.553926:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305951425 00000100:00000001:0.0:1787758223.553928:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305950730 00000100:00000001:0.0:1787758223.553929:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305951425 00000100:00000001:0.0:1787758223.553931:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758223.553932:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553933:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758223.553936:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758223/real 1787758223] req@ffff8ffb4a537800 x1874588659726720/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758223.553940:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758223.553941:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758223.553942:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758223.553943:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758223.553944:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553945:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758223.553946:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758223.553948:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659726720/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758223.553953:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553954:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758223.553956:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553957:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758223.553958:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.553959:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758223.553968:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758223.553970:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758223.553972:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758223.553974:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758223.553976:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758223.553978:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758223.553978:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758223.553979:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758223.553980:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758223.553981:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758223.553982:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758223.553984:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758223.553986:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758223.553987:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758223.553987:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758223.553988:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758223.553991:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.553993:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758223.553994:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758223.553997:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758223.554002:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758223.554004:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758223.554006:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758223.554007:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.554008:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758223.554009:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.554011:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305950730 00000100:00000001:0.0:1787758223.554013:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305951425 00000100:00000001:0.0:1787758223.554015:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305950730 00000100:00000001:0.0:1787758223.554016:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305951425 00000100:00000001:0.0:1787758223.554018:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758223.554019:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.554020:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758223.554022:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758223/real 1787758223] req@ffff8ffb4a537b80 x1874588659726848/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758223.554027:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758223.554028:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758223.554029:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758223.554030:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758223.554030:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554031:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758223.554032:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758223.554034:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659726848/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758223.554039:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554040:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758223.554042:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554043:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758223.554044:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554045:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758223.554053:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758223.554055:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758223.554056:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758223.554058:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758223.554060:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659726848:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758223.554062:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554064:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758223.554066:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758223.554067:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758223.554068:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758223.554069:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758223.554070:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758223.554071:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758223.554072:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758223.554081:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758223.554082:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554083:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758223.554084:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758223.554085:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758223.554087:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758223.554087:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758223.554088:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758223.554090:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758223.554091:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758223.554092:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758223.554092:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554093:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758223.554094:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758223.554095:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758223.554096:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758223.554097:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758223.554098:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758223.554098:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758223.554099:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758223.554100:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758223.554101:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554102:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758223.554104:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758223.554105:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758223.554106:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758223.554108:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758224.129268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758224.129274:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758224.129277:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758224.129280:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758224.129283:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758224.129287:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758224.129289:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758224.129291:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758224.129293:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758224.321278:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758224.321286:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758224.321292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758224.321296:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758224.321299:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758224.321303:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758224.321306:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758224.321308:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758224.321311:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758224.321313:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758224.321359:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758224.321362:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758224.321365:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758225.153246:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758225.153252:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758225.153254:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758225.153259:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758225.153262:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758225.153265:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758225.153267:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758225.153269:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758225.153271:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758225.345253:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758225.345263:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758225.345269:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758225.345270:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758225.345316:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758225.345321:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758225.345324:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758226.177252:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758226.177262:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758226.177264:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758226.177271:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758226.177276:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758226.177280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758226.177283:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758226.177287:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758226.177290:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758226.369302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758226.369308:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758226.369312:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758226.369313:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758226.369335:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758226.369338:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758226.369340:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758227.201259:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758227.201266:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758227.201268:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758227.201272:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758227.201275:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758227.201277:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758227.201279:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758227.201282:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758227.201284:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758227.393272:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758227.393278:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758227.393280:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758227.393282:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758227.393286:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758227.393288:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758227.393290:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758227.393291:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758227.393294:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758227.393295:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758227.393366:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758227.393369:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758227.393372:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758228.225304:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758228.225310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758228.225312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758228.225317:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758228.225320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758228.225323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758228.225325:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758228.225328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758228.225330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758228.417253:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758228.417259:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758228.417263:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758228.417264:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758228.417306:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758228.417310:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758228.417312:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758228.673301:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758228.673311:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758228.673313:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758228.673316:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758228.673321:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758228.673330:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11283 00000100:00080000:1.0:1787758228.673341:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758228.673345:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758228.673347:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758228.673352:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758228.673356:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758228.673360:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758228.673363:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758228.673366:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758228.673368:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758228.673373:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758228.673378:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758228.673381:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758228.673385:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758228.673399:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673401:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758228.673403:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758228.673414:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758228.673416:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673419:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758228.673421:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758228.673425:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673438:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758228.673446:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659726976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758228.673456:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758228.673459:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673474:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758228.673477:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758228.673480:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758228.673481:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758228.673483:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758228.673485:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758228.673487:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11283 00000100:00080000:1.0:1787758228.673490:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758228.673492:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758228.673493:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758228.673495:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758228.673496:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758228.673498:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758228.673500:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758228.673501:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758228.673502:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758228.673505:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758228.673507:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758228.673508:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758228.673513:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758228.673515:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673526:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758228.673531:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758228.673534:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673535:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758228.673539:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758228.673540:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787758228.673542:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673545:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758228.673546:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:1.0:1787758228.673547:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758228.673550:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758228.673550:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.673568:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758228.673570:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.673574:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758228.673576:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758228.673581:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673585:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758228.673588:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659726976:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1787758228.673601:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659727104/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758228.673608:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758228.673614:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758228.673614:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758228.673616:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.673618:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.673622:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758228.673622:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758228.673627:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758228.673630:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758228.673631:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758228.673635:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758228.673637:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758228.673640:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758228.673643:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11283 00000100:00080000:1.0:1787758228.673646:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758228.673647:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758228.673648:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758228.673651:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:1.0:1787758228.673652:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758228.673674:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758228.673675:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:1.0:1787758228.673676:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758228.673680:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758228.673682:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758228.673684:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758228.673686:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758228.673688:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758228.673691:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758228.673692:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758228.673695:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758228.673697:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673699:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758228.673714:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758228.673716:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673719:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758228.673723:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758228.673725:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673729:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000800:00000100:0.0:1787758228.673730:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:1.0:1787758228.673731:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758228.673735:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673741:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.673746:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:1.0:1787758228.673747:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659727232/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758228.673752:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758228.673756:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673758:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758228.673760:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758228.673763:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673768:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.673779:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758228.673781:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758228.673786:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758228.673787:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305956545 00000100:00000001:1.0:1787758228.673788:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758228.673791:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758228.673795:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758228.673798:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11283 00000100:00000001:0.0:1787758228.673800:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758228.673802:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758228.673806:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758228.673824:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305956545 00000100:00000001:1.0:1787758228.673824:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758228.673828:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758228.673830:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758228.673831:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:1.0:1787758228.673833:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758228.673837:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758228.673838:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758228.673839:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758228.673841:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758228.673843:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758228.673845:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758228.673848:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758228.673848:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758228.673850:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.673853:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.673857:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758228.673859:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758228.673863:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758228.673864:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758228/real 1787758228] req@ffff8ffb4a534a80 x1874588659726976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:1.0:1787758228.673868:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673872:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758228.673880:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:0.0:1787758228.673883:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:1.0:1787758228.673883:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758228.673887:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758228.673888:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758228.673902:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758228.673904:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.673905:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758228.673908:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.673910:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758228.673914:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.673918:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00100000:1.0:1787758228.673918:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659727360/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758228.673921:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758228.673931:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:0.0:1787758228.673932:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659726976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758228.673935:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758228.673939:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.673946:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.673950:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758228.673952:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.673965:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758228.673966:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.673968:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758228.673970:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758228.673972:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758228.673976:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758228.673978:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758228.673981:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659726976:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758228.673985:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.673987:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758228.673988:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758228.673990:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758228.673993:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758228.673994:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758228.673996:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758228.674000:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758228.674002:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758228.674004:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758228.674005:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674007:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674013:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758228.674014:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674016:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758228.674017:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758228.674019:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758228.674020:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758228.674021:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.674023:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758228.674024:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758228.674026:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758228.674028:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727104:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758228.674030:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758228.674032:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758228.674033:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.674034:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758228.674037:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674041:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758228.674042:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:0.0:1787758228.674046:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758228.674055:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758228.674059:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758228.674060:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758228.674061:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674062:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758228.674064:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674068:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305955853 00000100:00000001:0.0:1787758228.674070:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305956545 00000100:00000001:0.0:1787758228.674072:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305955853 00000100:00000001:0.0:1787758228.674074:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305956545 00000100:00000001:0.0:1787758228.674075:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758228.674076:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674078:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758228.674081:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758228/real 1787758228] req@ffff8ffb4a536300 x1874588659727104/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758228.674103:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758228.674105:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758228.674106:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758228.674108:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758228.674109:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674111:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758228.674112:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758228.674115:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659727104/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758228.674120:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674122:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758228.674124:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674125:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758228.674127:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674128:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758228.674139:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758228.674142:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758228.674144:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758228.674146:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758228.674149:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727104:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758228.674152:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758228.674153:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758228.674154:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.674156:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758228.674157:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758228.674159:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758228.674161:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758228.674165:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758228.674168:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758228.674170:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.674173:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758228.674177:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674184:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758228.674186:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758228.674190:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758228.674239:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758228.674245:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758228.674248:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758228.674249:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674251:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758228.674253:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674257:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305955851 00000100:00000001:0.0:1787758228.674260:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305956545 00000100:00000001:0.0:1787758228.674263:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305955851 00000100:00000001:0.0:1787758228.674266:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305956545 00000100:00000001:0.0:1787758228.674285:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758228.674288:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674290:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758228.674295:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758228/real 1787758228] req@ffff8ffb4a537b80 x1874588659727232/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758228.674304:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758228.674306:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758228.674309:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758228.674311:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758228.674313:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674315:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758228.674317:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758228.674322:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659727232/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758228.674330:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674333:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758228.674335:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674336:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758228.674337:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674338:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758228.674348:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758228.674349:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758228.674352:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758228.674354:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758228.674356:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758228.674359:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758228.674360:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758228.674361:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.674363:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758228.674365:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758228.674366:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758228.674369:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727360:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758228.674371:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758228.674372:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758228.674374:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758228.674375:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758228.674377:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674382:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758228.674383:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758228.674386:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758228.674396:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758228.674407:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758228.674409:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758228.674410:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674411:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758228.674413:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674416:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305955851 00000100:00000001:0.0:1787758228.674418:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305956545 00000100:00000001:0.0:1787758228.674419:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305955851 00000100:00000001:0.0:1787758228.674421:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305956545 00000100:00000001:0.0:1787758228.674423:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758228.674424:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674425:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758228.674428:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758228/real 1787758228] req@ffff8ffb4a537800 x1874588659727360/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758228.674433:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758228.674434:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758228.674436:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758228.674437:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758228.674438:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674439:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758228.674440:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758228.674443:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659727360/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758228.674453:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674454:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758228.674456:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674457:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758228.674458:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674459:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758228.674469:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758228.674472:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758228.674475:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758228.674478:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758228.674481:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727360:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758228.674485:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674487:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758228.674489:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758228.674490:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758228.674493:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758228.674494:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758228.674496:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758228.674498:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758228.674500:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758228.674502:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758228.674504:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674506:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758228.674507:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758228.674508:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758228.674511:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758228.674512:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758228.674514:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758228.674516:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758228.674518:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758228.674519:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758228.674521:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674523:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758228.674534:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758228.674535:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758228.674537:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758228.674538:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758228.674539:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758228.674541:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758228.674541:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758228.674542:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758228.674543:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674545:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758228.674548:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758228.674549:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758228.674550:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758228.674569:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758229.249270:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758229.249277:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758229.249279:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758229.249283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758229.249286:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758229.249289:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758229.249291:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758229.249293:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758229.249295:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758229.441286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758229.441294:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758229.441300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758229.441302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758229.441363:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758229.441366:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758229.441369:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758230.273335:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758230.273347:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758230.273349:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758230.273356:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758230.273359:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758230.273365:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758230.273368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758230.273371:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758230.273374:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758230.465304:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758230.465310:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758230.465315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758230.465316:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758230.465342:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758230.465345:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758230.465347:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758231.297298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758231.297307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758231.297309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758231.297315:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758231.297319:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758231.297323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758231.297326:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758231.297328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758231.297332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758231.489292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758231.489298:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758231.489303:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758231.489304:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758231.489407:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758231.489412:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758231.489416:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758232.321282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758232.321290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758232.321293:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758232.321299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758232.321303:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758232.321307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758232.321310:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758232.321313:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758232.321316:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758232.513284:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758232.513290:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758232.513295:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758232.513297:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758232.513379:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758232.513382:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758232.513385:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758233.345238:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758233.345243:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758233.345245:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758233.345250:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758233.345252:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758233.345256:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758233.345257:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758233.345259:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758233.345261:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758233.537271:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758233.537278:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758233.537284:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758233.537286:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758233.537372:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758233.537375:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758233.537377:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758233.793274:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758233.793290:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758233.793293:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758233.793298:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758233.793303:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758233.793312:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11288 00000100:00080000:1.0:1787758233.793326:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758233.793330:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758233.793332:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758233.793337:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758233.793340:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758233.793343:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758233.793345:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758233.793347:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758233.793348:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758233.793352:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758233.793354:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758233.793356:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758233.793360:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758233.793362:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793364:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758233.793367:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758233.793380:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758233.793382:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793385:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758233.793387:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758233.793392:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793404:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758233.793414:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659727488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758233.793423:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758233.793427:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793444:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758233.793447:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758233.793449:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758233.793450:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758233.793452:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758233.793454:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758233.793456:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11288 00000100:00080000:1.0:1787758233.793459:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758233.793460:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758233.793473:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758233.793476:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758233.793477:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758233.793478:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758233.793480:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758233.793481:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758233.793482:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758233.793485:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758233.793487:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758233.793488:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758233.793493:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758233.793494:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793496:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758233.793501:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758233.793503:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793505:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758233.793508:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758233.793510:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793511:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758233.793512:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758233.793513:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.793517:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758233.793519:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.793526:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758233.793531:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:1.0:1787758233.793532:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659727616/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758233.793534:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758233.793537:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758233.793545:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758233.793549:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758233.793556:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793556:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758233.793565:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758233.793569:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659727488:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758233.793574:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.793576:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787758233.793583:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:0.0:1787758233.793584:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758233.793590:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758233.793591:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:0.0:1787758233.793596:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758233.793596:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758233.793606:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758233.793610:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787758233.793611:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758233.793617:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11288 00000100:00080000:1.0:1787758233.793622:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758233.793625:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758233.793627:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758233.793628:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:0.0:1787758233.793630:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:1.0:1787758233.793630:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758233.793639:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758233.793640:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000020:00000001:1.0:1787758233.793654:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758233.793655:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758233.793660:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758233.793662:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758233.793666:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758233.793668:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758233.793670:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758233.793674:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758233.793676:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793677:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758233.793680:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758233.793683:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793685:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758233.793691:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758233.793692:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:0.0:1787758233.793697:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758233.793699:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758233.793701:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758233.793702:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793707:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758233.793713:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659727744/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758233.793719:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758233.793724:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758233.793725:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758233.793728:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.793729:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.793743:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758233.793743:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758233.793748:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:0.0:1787758233.793752:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758233.793752:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758233.793758:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758233.793761:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758233.793764:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305961665 00000100:00000001:1.0:1787758233.793766:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758233.793772:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11288 00000100:00080000:1.0:1787758233.793778:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758233.793779:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758233.793782:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758233.793786:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758233.793790:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305961665 00000020:00000001:1.0:1787758233.793790:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758233.793793:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758233.793797:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758233.793800:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:1.0:1787758233.793801:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758233.793808:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758233.793809:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758233.793813:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758233.793820:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758233.793821:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758233.793828:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758233.793828:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758233.793833:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758233.793837:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758233.793842:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758233.793844:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758233/real 1787758233] req@ffff8ffb4a534700 x1874588659727488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758233.793844:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758233.793849:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758233.793855:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793859:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758233.793866:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758233.793869:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793873:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758233.793875:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758233.793878:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758233.793885:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787758233.793887:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758233.793889:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758233.793897:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00100000:1.0:1787758233.793900:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659727872/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758233.793901:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758233.793904:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.793911:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758233.793915:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758233.793927:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659727488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758233.793933:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758233.793938:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758233.793947:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.793961:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.793965:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758233.793968:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.793970:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758233.793971:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.793973:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758233.793975:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758233.793978:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758233.793982:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758233.793984:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758233.793987:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659727488:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758233.793991:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.793993:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758233.793995:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758233.793997:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758233.794001:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758233.794003:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758233.794004:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758233.794008:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758233.794010:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758233.794012:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758233.794013:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794014:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794020:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758233.794021:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794024:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758233.794026:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758233.794027:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758233.794029:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758233.794030:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758233.794031:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758233.794033:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758233.794035:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758233.794038:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727616:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758233.794041:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758233.794042:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758233.794044:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758233.794045:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758233.794047:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794051:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758233.794052:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:0.0:1787758233.794056:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758233.794067:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758233.794070:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758233.794072:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758233.794073:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794074:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758233.794076:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794079:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305960973 00000100:00000001:0.0:1787758233.794081:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305961665 00000100:00000001:0.0:1787758233.794082:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305960973 00000100:00000001:0.0:1787758233.794084:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305961665 00000100:00000001:0.0:1787758233.794085:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758233.794086:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794087:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758233.794090:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758233/real 1787758233] req@ffff8ffb4a535c00 x1874588659727616/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758233.794095:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758233.794096:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758233.794098:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758233.794099:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758233.794100:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794101:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758233.794102:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758233.794104:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659727616/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758233.794109:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794110:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758233.794112:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794113:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758233.794114:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794115:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758233.794124:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758233.794126:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758233.794128:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758233.794129:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758233.794131:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727616:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758233.794134:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758233.794135:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758233.794136:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758233.794138:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758233.794139:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758233.794140:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758233.794142:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758233.794144:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758233.794145:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758233.794146:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758233.794147:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758233.794150:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794155:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758233.794157:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758233.794161:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758233.794172:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758233.794176:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758233.794179:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758233.794180:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794214:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758233.794218:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794222:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305960971 00000100:00000001:0.0:1787758233.794225:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305961665 00000100:00000001:0.0:1787758233.794227:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305960971 00000100:00000001:0.0:1787758233.794229:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305961665 00000100:00000001:0.0:1787758233.794232:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758233.794234:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794236:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758233.794241:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758233/real 1787758233] req@ffff8ffb4a537800 x1874588659727744/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758233.794250:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758233.794251:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758233.794254:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758233.794256:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758233.794257:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794260:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758233.794261:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758233.794265:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659727744/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758233.794273:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794276:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758233.794278:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794280:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758233.794281:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794284:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758233.794296:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758233.794299:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758233.794303:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758233.794306:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758233.794309:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758233.794313:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758233.794315:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758233.794316:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758233.794319:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758233.794322:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758233.794324:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758233.794327:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727872:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758233.794331:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758233.794333:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758233.794335:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758233.794337:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758233.794340:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794345:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758233.794347:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758233.794351:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758233.794364:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758233.794369:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758233.794371:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758233.794373:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794375:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758233.794377:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794381:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305960971 00000100:00000001:0.0:1787758233.794384:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305961665 00000100:00000001:0.0:1787758233.794386:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305960971 00000100:00000001:0.0:1787758233.794388:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305961665 00000100:00000001:0.0:1787758233.794391:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758233.794393:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794395:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758233.794399:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758233/real 1787758233] req@ffff8ffb4a537b80 x1874588659727872/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758233.794415:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758233.794416:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758233.794419:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758233.794420:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758233.794422:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794424:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758233.794425:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758233.794429:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659727872/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758233.794437:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794440:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758233.794442:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794444:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758233.794445:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794447:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758233.794459:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758233.794478:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758233.794482:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758233.794485:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758233.794488:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659727872:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758233.794492:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794494:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758233.794496:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758233.794497:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758233.794500:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758233.794502:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758233.794503:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758233.794506:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758233.794507:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758233.794510:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758233.794511:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794514:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758233.794515:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758233.794516:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758233.794518:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758233.794520:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758233.794521:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758233.794523:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758233.794524:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758233.794526:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758233.794527:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794529:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758233.794530:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758233.794531:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758233.794533:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758233.794534:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758233.794535:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758233.794537:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758233.794538:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758233.794540:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758233.794541:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794543:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758233.794546:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758233.794547:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758233.794549:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758233.794551:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758234.369282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758234.369293:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758234.369296:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758234.369304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758234.369308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758234.369313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758234.369316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758234.369320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758234.369323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758234.561334:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758234.561344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758234.561356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758234.561361:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758234.561364:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758234.561370:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758234.561374:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758234.561378:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758234.561380:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758234.561383:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758234.561412:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758234.561416:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758234.561419:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758235.393250:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758235.393257:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758235.393259:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758235.393265:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758235.393269:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758235.393273:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758235.393276:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758235.393279:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758235.393282:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758235.585277:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758235.585283:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758235.585289:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758235.585291:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758235.585378:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758235.585381:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758235.585384:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758236.417240:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758236.417246:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758236.417248:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758236.417252:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758236.417255:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758236.417258:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758236.417260:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758236.417263:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758236.417264:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758236.609273:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758236.609281:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758236.609288:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758236.609290:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758236.609325:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758236.609328:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758236.609331:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758237.441244:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758237.441251:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758237.441252:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758237.441256:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758237.441259:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758237.441262:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758237.441264:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758237.441266:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758237.441269:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758237.633257:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758237.633263:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758237.633265:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758237.633267:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758237.633271:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758237.633273:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758237.633274:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758237.633276:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758237.633279:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758237.633280:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758237.633321:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758237.633324:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758237.633327:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758238.465288:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758238.465296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758238.465298:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758238.465304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758238.465308:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758238.465312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758238.465315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758238.465318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758238.465322:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758238.657259:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758238.657264:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758238.657268:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758238.657269:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758238.657286:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758238.657288:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758238.657290:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758238.913269:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758238.913278:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758238.913281:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758238.913284:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758238.913288:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758238.913294:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11293 00000100:00080000:1.0:1787758238.913303:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758238.913305:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758238.913307:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758238.913311:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758238.913313:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758238.913316:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758238.913318:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758238.913320:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758238.913321:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758238.913326:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758238.913328:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758238.913330:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758238.913333:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758238.913336:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913338:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758238.913340:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758238.913348:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758238.913349:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913352:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758238.913354:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758238.913358:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913368:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758238.913389:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659728000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758238.913398:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758238.913402:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913416:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758238.913419:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758238.913421:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758238.913423:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758238.913425:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758238.913427:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758238.913430:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11293 00000100:00080000:1.0:1787758238.913434:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758238.913437:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758238.913439:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758238.913441:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758238.913442:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:1.0:1787758238.913443:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758238.913447:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758238.913448:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:1.0:1787758238.913450:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758238.913452:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758238.913453:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758238.913456:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758238.913456:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758238.913460:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758238.913462:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787758238.913466:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758238.913467:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758238.913471:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758238.913471:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758238.913476:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758238.913478:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758238.913483:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758238.913484:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659728000:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758238.913486:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758238.913491:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00100000:1.0:1787758238.913492:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787758238.913495:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00000001:1.0:1787758238.913496:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.913499:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913500:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.913503:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758238.913506:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758238.913509:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.913509:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913515:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758238.913516:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758238.913517:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913523:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913527:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:1.0:1787758238.913528:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659728128/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758238.913531:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787758238.913537:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758238.913541:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758238.913545:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913549:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758238.913552:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758238.913557:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758238.913559:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758238.913561:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758238.913565:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758238.913568:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11293 00000800:00000100:2.0:1787758238.913573:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787758238.913576:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758238.913581:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758238.913582:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758238.913583:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758238.913587:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:1.0:1787758238.913589:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758238.913590:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758238.913593:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787758238.913594:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:1.0:1787758238.913598:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758238.913601:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758238.913601:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913607:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758238.913608:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758238.913609:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305966785 00000100:00000001:2.0:1787758238.913617:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758238.913622:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758238.913623:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305966785 00000100:00000001:1.0:1787758238.913627:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913629:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787758238.913630:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758238.913635:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758238.913638:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913640:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758238.913640:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000008:00100000:1.0:1787758238.913645:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758238.913645:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787758238.913652:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913652:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758238.913655:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758238.913661:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00100000:2.0:1787758238.913661:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758238/real 1787758238] req@ffff8ffb4a536300 x1874588659728000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1787758238.913664:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913668:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758238.913671:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758238.913675:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787758238.913677:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758238.913680:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758238.913681:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913685:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00100000:1.0:1787758238.913689:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659728256/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758238.913689:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758238.913694:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.913699:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758238.913701:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758238.913702:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758238.913705:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913708:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758238.913710:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659728000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758238.913711:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758238.913718:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758238.913721:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758238.913724:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758238.913729:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787758238.913730:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758238.913734:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11293 00000100:00000001:2.0:1787758238.913734:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758238.913740:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758238.913741:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758238.913742:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758238.913745:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758238.913746:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758238.913748:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758238.913749:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758238.913751:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:1.0:1787758238.913754:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00080000:2.0:1787758238.913756:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000020:00000001:1.0:1787758238.913757:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758238.913761:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:2.0:1787758238.913764:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000020:00000001:1.0:1787758238.913765:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758238.913769:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1787758238.913770:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758238.913772:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787758238.913776:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659728000:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787758238.913779:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758238.913784:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913784:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758238.913790:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758238.913790:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758238.913794:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758238.913795:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758238.913798:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.913798:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:1.0:1787758238.913801:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:2.0:1787758238.913802:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000008:00100000:1.0:1787758238.913803:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000020:00000001:2.0:1787758238.913806:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000008:00000001:1.0:1787758238.913807:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758238.913808:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:1.0:1787758238.913810:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.913814:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758238.913815:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758238.913816:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 02000000:00000001:1.0:1787758238.913817:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758238.913821:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787758238.913821:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 02000000:00000001:1.0:1787758238.913825:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758238.913825:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758238.913828:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913829:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758238.913833:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913836:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758238.913840:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758238.913841:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659728384/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758238.913843:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758238.913847:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758238.913850:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758238.913851:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758238.913852:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758238.913854:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913855:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.913859:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758238.913860:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.913861:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758238.913865:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758238.913869:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728128:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758238.913871:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758238.913872:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758238.913874:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.913875:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758238.913879:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913884:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758238.913886:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787758238.913889:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758238.913901:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758238.913907:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758238.913910:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758238.913912:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913913:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758238.913918:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913921:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305966093 00000100:00000001:2.0:1787758238.913924:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305966785 00000100:00000001:2.0:1787758238.913926:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305966093 00000100:00000001:2.0:1787758238.913928:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305966785 00000100:00000001:2.0:1787758238.913930:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758238.913931:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.913933:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758238.913936:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758238/real 1787758238] req@ffff8ffb4a534a80 x1874588659728128/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758238.913942:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758238.913943:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758238.913954:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758238.913955:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758238.913956:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.913957:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758238.913959:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758238.913961:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659728128/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758238.913970:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.913972:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758238.913974:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.913975:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758238.913975:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.913976:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758238.913987:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758238.913988:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758238.913990:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758238.913991:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758238.913993:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728128:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758238.913996:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758238.913997:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758238.913998:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.914000:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758238.914001:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758238.914002:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758238.914004:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758238.914006:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758238.914007:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758238.914008:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.914009:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758238.914012:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914015:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758238.914016:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787758238.914018:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758238.914025:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758238.914028:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758238.914029:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758238.914030:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914032:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758238.914033:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914035:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305966090 00000100:00000001:2.0:1787758238.914037:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305966785 00000100:00000001:2.0:1787758238.914039:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305966090 00000100:00000001:2.0:1787758238.914040:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305966785 00000100:00000001:2.0:1787758238.914042:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758238.914043:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914044:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758238.914046:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758238/real 1787758238] req@ffff8ffb4a537b80 x1874588659728256/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758238.914052:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758238.914053:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758238.914055:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758238.914057:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758238.914058:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914060:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758238.914061:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758238.914065:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659728256/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758238.914072:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914074:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758238.914077:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914078:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758238.914079:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914082:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758238.914093:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758238.914096:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758238.914100:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758238.914102:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758238.914106:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758238.914110:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914112:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758238.914113:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758238.914114:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758238.914116:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758238.914117:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758238.914118:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758238.914120:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758238.914121:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758238.914122:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758238.914123:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914124:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758238.914125:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758238.914126:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758238.914127:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758238.914127:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758238.914128:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758238.914129:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758238.914130:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758238.914131:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758238.914131:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914132:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914135:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758238.914143:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914145:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758238.914145:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758238.914146:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758238.914147:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758238.914148:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.914150:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758238.914151:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758238.914152:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758238.914155:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728384:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758238.914157:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758238.914158:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758238.914159:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758238.914161:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758238.914163:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914167:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758238.914168:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787758238.914171:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758238.914180:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758238.914204:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758238.914206:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758238.914208:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914209:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758238.914211:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914214:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305966091 00000100:00000001:2.0:1787758238.914216:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305966785 00000100:00000001:2.0:1787758238.914219:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305966091 00000100:00000001:2.0:1787758238.914221:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305966785 00000100:00000001:2.0:1787758238.914222:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758238.914224:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914225:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758238.914228:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758238/real 1787758238] req@ffff8ffb4a537800 x1874588659728384/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758238.914234:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758238.914235:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758238.914237:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758238.914239:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758238.914241:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914243:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758238.914244:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758238.914248:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659728384/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758238.914253:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914255:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758238.914256:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914257:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758238.914258:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914260:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758238.914268:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758238.914270:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758238.914273:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758238.914274:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758238.914277:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728384:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758238.914279:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914280:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758238.914280:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758238.914281:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758238.914283:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758238.914283:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758238.914284:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758238.914285:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758238.914286:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758238.914288:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758238.914288:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914290:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758238.914292:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758238.914293:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758238.914294:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758238.914295:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758239.489291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758239.489298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758239.489300:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758239.489306:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758239.489309:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758239.489313:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758239.489317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758239.489320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758239.489322:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758239.681295:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758239.681317:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758239.681325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758239.681328:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758239.681398:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758239.681401:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758239.681403:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758240.513312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758240.513321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758240.513324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758240.513331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758240.513336:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758240.513341:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758240.513399:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758240.513403:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758240.513406:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758240.705401:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758240.705409:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758240.705417:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758240.705419:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758240.705510:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758240.705516:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758240.705521:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758241.537259:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758241.537267:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758241.537269:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758241.537274:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758241.537277:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758241.537280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758241.537283:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758241.537285:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758241.537287:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758241.729311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758241.729342:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758241.729354:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758241.729356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758241.729459:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758241.729466:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758241.729472:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758242.561291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758242.561301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758242.561304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758242.561333:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758242.561339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758242.561347:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758242.561350:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758242.561354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758242.561358:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758242.753345:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758242.753354:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758242.753363:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758242.753365:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758242.753395:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758242.753398:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758242.753402:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758243.585390:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758243.585399:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758243.585402:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758243.585408:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758243.585412:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758243.585417:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758243.585420:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758243.585423:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758243.585426:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758243.777407:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758243.777419:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758243.777428:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758243.777430:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758243.777461:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758243.777464:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758243.777468:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758244.033385:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758244.033394:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758244.033397:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758244.033400:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758244.033403:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758244.033410:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11298 00000100:00080000:2.0:1787758244.033420:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758244.033423:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758244.033425:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758244.033429:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758244.033431:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758244.033434:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758244.033436:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758244.033437:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758244.033438:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758244.033441:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758244.033443:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758244.033445:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758244.033448:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758244.033451:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033453:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758244.033455:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758244.033465:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758244.033466:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033470:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758244.033472:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758244.033476:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033487:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758244.033494:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659728512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758244.033503:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758244.033506:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033523:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758244.033526:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758244.033529:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758244.033530:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758244.033532:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758244.033534:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758244.033536:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11298 00000100:00080000:2.0:1787758244.033539:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758244.033541:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758244.033542:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758244.033544:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758244.033545:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758244.033546:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758244.033548:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758244.033549:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758244.033550:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758244.033553:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758244.033554:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758244.033556:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758244.033560:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758244.033562:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033564:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758244.033569:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758244.033571:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033573:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758244.033576:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758244.033577:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033579:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758244.033579:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758244.033580:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758244.033585:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033586:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758244.033589:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033591:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758244.033595:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00100000:2.0:1787758244.033597:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659728640/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758244.033599:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.033608:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787758244.033608:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758244.033612:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.033614:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758244.033615:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.033624:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758244.033624:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758244.033630:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758244.033632:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758244.033634:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758244.033637:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659728512:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758244.033638:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758244.033643:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11298 00000100:00000001:0.0:1787758244.033649:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787758244.033649:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758244.033653:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758244.033654:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:0.0:1787758244.033656:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:2.0:1787758244.033656:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758244.033660:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:0.0:1787758244.033661:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758244.033662:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758244.033667:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:0.0:1787758244.033668:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758244.033668:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758244.033672:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758244.033675:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787758244.033677:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033678:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758244.033684:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758244.033686:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758244.033687:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033688:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758244.033690:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758244.033692:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033693:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758244.033697:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758244.033698:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033699:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758244.033700:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:2.0:1787758244.033701:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758244.033706:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 02000000:00000001:2.0:1787758244.033706:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033712:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033714:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:2.0:1787758244.033719:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659728768/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758244.033725:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758244.033727:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033729:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758244.033731:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758244.033734:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758244.033736:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758244.033737:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758244.033739:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758244.033741:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11298 00000100:00080000:2.0:1787758244.033750:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758244.033752:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758244.033753:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758244.033754:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758244.033756:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758244.033758:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000800:00000100:0.0:1787758244.033761:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787758244.033762:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758244.033766:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758244.033768:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758244.033771:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758244.033774:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758244.033775:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758244.033777:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758244.033779:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033780:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758244.033780:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758244.033785:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758244.033787:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00000001:2.0:1787758244.033787:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033792:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033793:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758244.033796:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758244.033798:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787758244.033799:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033802:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758244.033803:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758244.033805:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033806:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033809:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033813:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305971905 00000100:00100000:2.0:1787758244.033814:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659728896/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758244.033829:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758244.033830:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758244.033831:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758244.033837:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033842:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305971905 00000100:00000001:0.0:1787758244.033850:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758244.033856:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758244.033860:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033863:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758244.033871:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758244/real 1787758244] req@ffff8ffb47a38a80 x1874588659728512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758244.033884:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758244.033886:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758244.033890:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758244.033893:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758244.033895:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.033898:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758244.033901:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758244.033906:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659728512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758244.033916:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.033919:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758244.033923:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.033925:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758244.033926:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.033929:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758244.033931:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758244.033934:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758244.033938:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758244.033941:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758244.033946:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659728512:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758244.033952:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.033954:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758244.033957:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758244.033960:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758244.033964:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758244.033967:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758244.033970:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758244.033973:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758244.033975:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758244.033979:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758244.033981:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.033983:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.033991:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758244.033994:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.033996:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758244.033998:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758244.034000:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758244.034001:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758244.034003:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.034006:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758244.034008:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758244.034011:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758244.034015:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728640:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758244.034019:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758244.034021:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758244.034023:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.034025:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758244.034029:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034035:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758244.034037:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:0.0:1787758244.034042:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758244.034055:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758244.034061:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758244.034064:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758244.034065:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034067:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758244.034071:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034076:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305971213 00000100:00000001:0.0:1787758244.034079:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305971905 00000100:00000001:0.0:1787758244.034082:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305971213 00000100:00000001:0.0:1787758244.034085:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305971905 00000100:00000001:0.0:1787758244.034088:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758244.034090:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034092:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758244.034097:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758244/real 1787758244] req@ffff8ffb47a39500 x1874588659728640/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758244.034108:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758244.034110:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758244.034113:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758244.034115:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758244.034117:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034119:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758244.034120:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758244.034125:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659728640/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758244.034133:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034136:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758244.034139:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034141:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758244.034142:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034144:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758244.034159:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758244.034162:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758244.034166:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758244.034169:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758244.034173:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728640:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758244.034177:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758244.034179:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758244.034207:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.034210:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758244.034213:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758244.034216:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758244.034220:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758244.034224:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758244.034226:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758244.034228:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.034230:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758244.034234:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034240:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758244.034242:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:0.0:1787758244.034246:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758244.034260:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758244.034265:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758244.034267:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758244.034269:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034271:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758244.034274:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034278:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305971211 00000100:00000001:0.0:1787758244.034281:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305971905 00000100:00000001:0.0:1787758244.034298:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305971211 00000100:00000001:0.0:1787758244.034301:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305971905 00000100:00000001:0.0:1787758244.034304:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758244.034306:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034309:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758244.034314:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758244/real 1787758244] req@ffff8ffb47a3a680 x1874588659728768/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758244.034325:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758244.034326:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758244.034330:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758244.034332:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758244.034334:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034336:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758244.034337:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758244.034343:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659728768/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758244.034351:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034354:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758244.034357:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034359:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758244.034361:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034363:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758244.034376:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758244.034380:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758244.034385:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758244.034388:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758244.034392:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758244.034397:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758244.034399:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758244.034401:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.034404:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758244.034406:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758244.034408:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758244.034412:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758244.034415:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758244.034418:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758244.034420:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758244.034422:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758244.034425:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034432:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758244.034433:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:0.0:1787758244.034437:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758244.034451:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758244.034457:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758244.034459:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758244.034461:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034463:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758244.034467:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034470:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305971211 00000100:00000001:0.0:1787758244.034473:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305971905 00000100:00000001:0.0:1787758244.034476:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305971211 00000100:00000001:0.0:1787758244.034479:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305971905 00000100:00000001:0.0:1787758244.034481:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758244.034483:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034485:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758244.034490:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758244/real 1787758244] req@ffff8ffb47a39880 x1874588659728896/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758244.034500:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758244.034501:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758244.034504:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758244.034506:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758244.034507:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034510:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758244.034511:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758244.034515:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659728896/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758244.034523:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034526:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758244.034529:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034531:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758244.034532:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034534:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758244.034564:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758244.034568:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758244.034572:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758244.034575:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758244.034579:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659728896:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758244.034583:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034586:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758244.034588:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758244.034590:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758244.034593:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758244.034594:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758244.034596:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758244.034599:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758244.034600:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758244.034604:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758244.034605:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034607:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758244.034609:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758244.034611:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758244.034614:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758244.034615:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758244.034616:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758244.034619:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758244.034620:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758244.034623:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758244.034624:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034626:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758244.034627:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758244.034629:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758244.034631:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758244.034632:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758244.034633:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758244.034636:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758244.034637:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758244.034639:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758244.034640:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034643:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758244.034647:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758244.034649:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758244.034651:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758244.034653:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758244.609317:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758244.609325:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758244.609327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758244.609333:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758244.609336:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758244.609340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758244.609342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758244.609345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758244.609348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758244.801338:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758244.801347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758244.801356:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758244.801361:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758244.801365:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758244.801368:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758244.801372:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758244.801375:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758244.801377:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758244.801379:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758244.801429:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758244.801452:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758244.801457:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758245.633348:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758245.633357:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758245.633359:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758245.633364:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758245.633367:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758245.633371:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758245.633373:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758245.633375:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758245.633378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758245.825292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758245.825297:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758245.825302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758245.825303:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758245.825365:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758245.825368:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758245.825371:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758246.657293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758246.657301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758246.657303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758246.657308:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758246.657311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758246.657315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758246.657318:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758246.657320:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758246.657323:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758246.849306:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758246.849315:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758246.849322:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758246.849324:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758246.849371:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758246.849373:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758246.849376:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758247.681271:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758247.681277:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758247.681279:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758247.681283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758247.681286:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758247.681289:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758247.681291:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758247.681294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758247.681296:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758247.873330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758247.873338:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758247.873340:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758247.873342:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758247.873347:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758247.873349:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758247.873350:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758247.873352:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758247.873355:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758247.873356:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758247.873383:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758247.873385:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758247.873387:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758248.705326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758248.705335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758248.705338:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758248.705344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758248.705348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758248.705354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758248.705357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758248.705361:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758248.705364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758248.897313:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758248.897321:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758248.897327:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758248.897329:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758248.897395:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758248.897400:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758248.897404:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758249.153290:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758249.153299:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758249.153302:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758249.153305:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758249.153309:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758249.153315:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11303 00000100:00080000:1.0:1787758249.153325:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758249.153328:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758249.153330:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758249.153334:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758249.153336:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758249.153339:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758249.153341:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758249.153343:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758249.153345:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758249.153349:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758249.153351:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.153354:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758249.153357:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758249.153360:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153362:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758249.153364:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.153373:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758249.153374:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153377:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758249.153380:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758249.153383:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153394:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758249.153403:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659729024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758249.153412:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758249.153416:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153434:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758249.153438:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758249.153445:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758249.153447:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758249.153449:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758249.153453:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758249.153456:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11303 00000100:00080000:1.0:1787758249.153461:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758249.153464:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758249.153466:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758249.153470:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758249.153473:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758249.153476:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758249.153479:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758249.153482:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758249.153484:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758249.153486:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787758249.153491:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758249.153492:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758249.153496:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758249.153497:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153500:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787758249.153501:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758249.153504:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.153508:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758249.153528:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758249.153531:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.153533:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758249.153535:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787758249.153539:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:1.0:1787758249.153544:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758249.153549:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659729024:192.168.201.118@tcp:250:kworker.0 00000008:00000001:1.0:1787758249.153550:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153558:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153560:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758249.153565:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758249.153566:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758249.153570:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.153571:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153574:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758249.153577:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758249.153578:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758249.153583:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.153585:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153592:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758249.153602:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659729152/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758249.153606:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758249.153611:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:0.0:1787758249.153618:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758249.153628:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758249.153632:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153637:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758249.153640:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758249.153644:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758249.153647:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758249.153649:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758249.153652:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758249.153655:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11303 00000100:00080000:1.0:1787758249.153660:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758249.153662:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758249.153666:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:0.0:1787758249.153667:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:1.0:1787758249.153673:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758249.153676:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758249.153679:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758249.153683:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758249.153684:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758249.153686:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758249.153690:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758249.153692:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758249.153698:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758249.153698:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758249.153707:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758249.153707:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.153712:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758249.153714:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.153716:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758249.153721:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305977025 02000000:00000001:1.0:1787758249.153722:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153730:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758249.153733:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758249.153736:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153738:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758249.153739:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153748:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305977025 02000000:00000001:1.0:1787758249.153750:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758249.153754:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153759:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758249.153759:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758249.153774:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758249.153775:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758249.153781:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.153782:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153789:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758249.153794:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758249.153800:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758249/real 1787758249] req@ffff8ffb4a535c00 x1874588659729024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758249.153825:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659729280/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:0.0:1787758249.153839:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758249.153841:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758249.153846:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758249.153848:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758249.153851:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758249.153852:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758249.153854:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758249.153858:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.153863:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153866:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758249.153867:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758249.153874:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659729024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758249.153876:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758249.153881:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758249.153885:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758249.153889:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758249.153892:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:0.0:1787758249.153893:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758249.153898:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.153901:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:1.0:1787758249.153901:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11303 00000100:00000001:0.0:1787758249.153904:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.153909:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00080000:1.0:1787758249.153910:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758249.153911:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758249.153915:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00080000:0.0:1787758249.153917:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:1.0:1787758249.153919:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758249.153924:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000020:00000001:1.0:1787758249.153925:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758249.153929:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:0.0:1787758249.153930:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:1.0:1787758249.153933:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:0.0:1787758249.153936:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659729024:192.168.201.118@tcp:250:kworker.0 00000020:00000001:1.0:1787758249.153937:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758249.153944:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758249.153944:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758249.153946:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:0.0:1787758249.153948:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758249.153951:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00080000:1.0:1787758249.153955:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:0.0:1787758249.153956:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758249.153961:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:1.0:1787758249.153961:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.153964:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000020:00000001:0.0:1787758249.153966:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758249.153969:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758249.153969:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758249.153974:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.153976:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:1.0:1787758249.153976:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:0.0:1787758249.153980:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000008:00100000:1.0:1787758249.153981:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758249.153984:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000008:00000001:1.0:1787758249.153986:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153987:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.153990:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758249.153990:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.153996:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758249.153998:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.153999:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758249.154000:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758249.154002:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758249.154003:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758249.154006:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787758249.154007:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154009:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758249.154012:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758249.154014:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154015:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758249.154017:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.154021:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:1.0:1787758249.154024:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659729408/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:0.0:1787758249.154025:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758249.154030:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758249.154037:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729152:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758249.154044:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787758249.154046:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:0.0:1787758249.154048:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758249.154051:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.154052:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.154055:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758249.154058:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.154059:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154069:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758249.154071:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:0.0:1787758249.154075:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758249.154122:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758249.154133:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758249.154136:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758249.154137:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154139:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758249.154142:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154147:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305976333 00000100:00000001:0.0:1787758249.154150:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305977025 00000100:00000001:0.0:1787758249.154153:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305976333 00000100:00000001:0.0:1787758249.154155:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305977025 00000100:00000001:0.0:1787758249.154157:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758249.154159:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154161:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758249.154166:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758249/real 1787758249] req@ffff8ffb4a534700 x1874588659729152/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758249.154175:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758249.154176:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758249.154178:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758249.154180:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758249.154240:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154243:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758249.154244:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758249.154251:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659729152/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758249.154260:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154262:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758249.154266:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154268:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758249.154270:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154272:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758249.154287:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758249.154290:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758249.154295:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758249.154298:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758249.154302:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729152:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758249.154307:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758249.154308:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758249.154310:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.154312:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758249.154313:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758249.154315:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758249.154318:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758249.154320:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758249.154322:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758249.154323:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.154325:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758249.154328:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154333:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758249.154334:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:0.0:1787758249.154338:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758249.154349:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758249.154353:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758249.154355:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758249.154357:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154358:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758249.154360:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154363:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305976331 00000100:00000001:0.0:1787758249.154365:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305977025 00000100:00000001:0.0:1787758249.154367:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305976331 00000100:00000001:0.0:1787758249.154369:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305977025 00000100:00000001:0.0:1787758249.154370:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758249.154372:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154373:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758249.154376:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758249/real 1787758249] req@ffff8ffb4a537800 x1874588659729280/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758249.154382:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758249.154383:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758249.154385:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758249.154386:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758249.154388:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154389:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758249.154390:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758249.154393:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659729280/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758249.154399:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154400:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758249.154402:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154403:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758249.154404:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154406:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758249.154414:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758249.154416:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758249.154419:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758249.154420:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758249.154423:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758249.154426:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154427:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758249.154428:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758249.154430:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758249.154431:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758249.154432:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758249.154433:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758249.154435:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758249.154436:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758249.154438:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758249.154439:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154440:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758249.154441:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758249.154442:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758249.154444:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758249.154445:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758249.154446:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758249.154448:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758249.154448:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758249.154450:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758249.154451:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154452:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154455:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758249.154456:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154457:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758249.154458:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758249.154459:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758249.154460:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758249.154461:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.154463:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758249.154464:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758249.154465:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758249.154467:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758249.154470:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758249.154471:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758249.154472:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758249.154474:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758249.154476:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154480:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758249.154481:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:0.0:1787758249.154484:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758249.154492:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758249.154511:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758249.154513:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758249.154515:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154517:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758249.154519:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154522:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305976331 00000100:00000001:0.0:1787758249.154525:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305977025 00000100:00000001:0.0:1787758249.154527:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305976331 00000100:00000001:0.0:1787758249.154530:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305977025 00000100:00000001:0.0:1787758249.154532:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758249.154534:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154536:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758249.154540:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758249/real 1787758249] req@ffff8ffb4a537b80 x1874588659729408/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758249.154547:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758249.154549:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758249.154551:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758249.154552:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758249.154554:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154556:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758249.154557:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758249.154560:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659729408/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758249.154568:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154570:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758249.154572:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154574:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758249.154576:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154577:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758249.154588:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758249.154591:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758249.154594:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758249.154597:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758249.154600:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729408:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758249.154603:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154605:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758249.154606:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758249.154607:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758249.154609:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758249.154610:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758249.154612:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758249.154613:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758249.154615:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758249.154616:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758249.154618:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154620:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758249.154622:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758249.154624:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758249.154626:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758249.154628:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758249.729345:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758249.729354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758249.729357:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758249.729363:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758249.729368:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758249.729372:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758249.729376:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758249.729379:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758249.729383:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758249.921357:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758249.921368:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758249.921383:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758249.921387:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758249.921474:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758249.921485:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758249.921504:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758250.753291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758250.753299:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758250.753301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758250.753307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758250.753310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758250.753314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758250.753317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758250.753319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758250.753322:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758250.945293:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758250.945303:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758250.945311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758250.945313:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758250.945342:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758250.945346:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758250.945350:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758251.777348:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758251.777359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758251.777362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758251.777367:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758251.777371:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758251.777376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758251.777378:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758251.777381:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758251.777383:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758251.969293:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758251.969299:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758251.969304:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758251.969305:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758251.969325:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758251.969328:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758251.969330:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758252.801386:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758252.801394:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758252.801396:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758252.801401:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758252.801404:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758252.801408:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758252.801411:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758252.801415:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758252.801420:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758252.993288:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758252.993296:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758252.993303:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758252.993306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758252.993373:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758252.993378:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758252.993401:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758253.825291:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758253.825298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758253.825299:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758253.825304:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758253.825307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758253.825311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758253.825313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758253.825315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758253.825317:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758254.017269:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758254.017275:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758254.017280:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758254.017282:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758254.017306:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758254.017309:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758254.017312:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758254.273303:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758254.273311:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758254.273313:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758254.273316:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758254.273320:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758254.273326:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11309 00000100:00080000:2.0:1787758254.273336:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758254.273338:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758254.273340:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758254.273343:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758254.273345:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758254.273347:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758254.273349:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758254.273350:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758254.273351:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758254.273354:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758254.273356:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758254.273358:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758254.273361:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758254.273363:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273365:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758254.273366:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758254.273376:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758254.273377:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273380:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758254.273382:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758254.273385:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273404:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758254.273411:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659729536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758254.273419:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758254.273422:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273433:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758254.273436:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758254.273438:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758254.273439:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758254.273441:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758254.273443:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758254.273445:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11309 00000100:00080000:2.0:1787758254.273447:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758254.273449:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758254.273450:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758254.273452:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758254.273452:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758254.273454:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758254.273455:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758254.273457:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758254.273457:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758254.273459:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758254.273461:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758254.273462:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758254.273466:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758254.273467:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273468:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758254.273473:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758254.273476:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273477:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758254.273480:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758254.273481:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273482:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758254.273483:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758254.273485:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787758254.273485:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273491:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758254.273492:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273494:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758254.273498:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00100000:2.0:1787758254.273499:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659729664/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758254.273500:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.273508:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787758254.273509:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758254.273513:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.273514:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758254.273516:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.273521:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758254.273522:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758254.273531:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659729536:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787758254.273533:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758254.273537:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:0.0:1787758254.273541:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787758254.273541:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758254.273543:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:0.0:1787758254.273547:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:2.0:1787758254.273548:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11309 02000000:00000001:0.0:1787758254.273553:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758254.273553:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:0.0:1787758254.273558:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758254.273558:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758254.273562:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758254.273563:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:0.0:1787758254.273565:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758254.273565:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758254.273569:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758254.273572:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758254.273573:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758254.273575:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758254.273577:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758254.273578:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758254.273579:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758254.273581:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758254.273582:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273584:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758254.273585:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758254.273587:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273588:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273592:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:2.0:1787758254.273592:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:0.0:1787758254.273595:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 02000000:00000001:2.0:1787758254.273596:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273600:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:0.0:1787758254.273602:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787758254.273602:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758254.273604:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273611:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758254.273614:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659729792/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758254.273624:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758254.273626:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273628:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758254.273641:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758254.273644:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758254.273645:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758254.273646:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758254.273648:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758254.273650:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11309 00000100:00080000:2.0:1787758254.273652:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758254.273654:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758254.273655:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758254.273659:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000800:00000100:0.0:1787758254.273660:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000020:00000001:2.0:1787758254.273661:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758254.273664:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758254.273666:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758254.273668:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758254.273669:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758254.273673:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758254.273674:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758254.273675:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:0.0:1787758254.273677:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787758254.273678:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758254.273682:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273684:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758254.273685:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00100000:2.0:1787758254.273686:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:0.0:1787758254.273689:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787758254.273689:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273694:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758254.273695:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758254.273700:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758254.273701:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273702:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273703:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758254.273706:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:0.0:1787758254.273707:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305982145 02000000:00000001:2.0:1787758254.273708:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273716:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273719:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:2.0:1787758254.273724:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659729920/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758254.273726:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305982145 00000100:00000001:0.0:1787758254.273732:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758254.273737:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758254.273743:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.273744:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758254.273745:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758254.273750:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758254.273753:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758254/real 1787758254] req@ffff8ffb47a38380 x1874588659729536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758254.273755:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1787758254.273776:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758254.273778:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758254.273781:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758254.273782:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758254.273784:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273786:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758254.273788:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758254.273791:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659729536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758254.273797:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273800:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758254.273802:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273803:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758254.273804:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273806:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758254.273807:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758254.273809:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758254.273812:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758254.273814:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758254.273816:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659729536:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758254.273819:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273821:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758254.273822:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758254.273824:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758254.273827:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758254.273829:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758254.273831:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758254.273834:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758254.273835:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758254.273837:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758254.273838:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273840:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273846:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758254.273847:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273849:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758254.273850:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758254.273851:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758254.273852:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758254.273853:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.273855:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758254.273856:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758254.273857:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758254.273859:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729664:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758254.273862:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758254.273863:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758254.273864:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.273865:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758254.273867:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273871:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758254.273872:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:0.0:1787758254.273876:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758254.273885:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758254.273888:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758254.273889:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758254.273890:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273891:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758254.273893:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273896:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305981453 00000100:00000001:0.0:1787758254.273898:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305982145 00000100:00000001:0.0:1787758254.273900:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305981453 00000100:00000001:0.0:1787758254.273902:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305982145 00000100:00000001:0.0:1787758254.273903:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758254.273905:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273906:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758254.273908:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758254/real 1787758254] req@ffff8ffb47a38000 x1874588659729664/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758254.273914:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758254.273915:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758254.273916:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758254.273917:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758254.273918:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273919:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758254.273920:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758254.273923:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659729664/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758254.273927:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273928:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758254.273930:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273931:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758254.273932:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.273933:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758254.273942:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758254.273944:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758254.273946:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758254.273947:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758254.273949:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729664:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758254.273952:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758254.273953:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758254.273954:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.273955:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758254.273956:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758254.273958:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758254.273960:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729792:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758254.273962:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758254.273963:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758254.273963:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.273965:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758254.273966:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.273969:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758254.273970:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:0.0:1787758254.273974:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758254.274002:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758254.274007:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758254.274009:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758254.274011:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.274013:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758254.274023:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.274027:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305981450 00000100:00000001:0.0:1787758254.274030:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305982145 00000100:00000001:0.0:1787758254.274032:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305981450 00000100:00000001:0.0:1787758254.274033:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305982145 00000100:00000001:0.0:1787758254.274035:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758254.274036:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.274037:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758254.274041:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758254/real 1787758254] req@ffff8ffb47a3b480 x1874588659729792/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758254.274046:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758254.274047:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758254.274049:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758254.274050:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758254.274051:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274052:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758254.274053:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758254.274056:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659729792/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758254.274061:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274062:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758254.274064:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274065:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758254.274065:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274066:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758254.274075:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758254.274076:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758254.274078:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758254.274079:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758254.274081:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729792:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758254.274083:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758254.274090:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758254.274091:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.274092:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758254.274094:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758254.274095:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758254.274097:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758254.274099:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:0.0:1787758254.274100:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:0.0:1787758254.274115:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758254.274117:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758254.274120:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.274123:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758254.274124:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:0.0:1787758254.274127:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:0.0:1787758254.274136:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758254.274140:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758254.274141:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758254.274142:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.274144:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758254.274145:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.274148:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305981450 00000100:00000001:0.0:1787758254.274149:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305982145 00000100:00000001:0.0:1787758254.274151:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305981450 00000100:00000001:0.0:1787758254.274152:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305982145 00000100:00000001:0.0:1787758254.274154:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758254.274155:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.274156:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:0.0:1787758254.274159:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758254/real 1787758254] req@ffff8ffb47a39180 x1874588659729920/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:0.0:1787758254.274163:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:0.0:1787758254.274164:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758254.274166:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758254.274167:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758254.274168:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274169:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758254.274170:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:0.0:1787758254.274172:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659729920/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758254.274177:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274178:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758254.274180:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274202:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:0.0:1787758254.274203:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274204:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758254.274213:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:0.0:1787758254.274214:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758254.274217:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758254.274218:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:0.0:1787758254.274221:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659729920:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758254.274223:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274224:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758254.274225:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758254.274227:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758254.274228:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758254.274229:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758254.274230:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758254.274232:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758254.274233:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758254.274235:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758254.274236:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274237:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758254.274237:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758254.274238:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758254.274239:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758254.274240:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758254.274241:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758254.274241:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758254.274242:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758254.274244:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758254.274244:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274245:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758254.274246:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758254.274247:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:0.0:1787758254.274248:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1787758254.274249:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:0.0:1787758254.274250:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:0.0:1787758254.274251:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1787758254.274252:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758254.274254:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1787758254.274255:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274256:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758254.274258:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1787758254.274259:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758254.274260:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758254.274262:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758254.849285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758254.849294:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758254.849297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758254.849302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758254.849306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758254.849311:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758254.849314:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758254.849317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758254.849320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758255.041369:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758255.041378:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758255.041384:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758255.041389:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758255.041393:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758255.041397:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758255.041401:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758255.041404:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758255.041407:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758255.041410:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758255.041434:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758255.041437:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758255.041441:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758255.873285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758255.873292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758255.873295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758255.873300:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758255.873303:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758255.873307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758255.873309:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758255.873312:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758255.873315:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758256.065291:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758256.065295:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758256.065300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758256.065302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758256.065322:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758256.065324:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758256.065326:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758256.897265:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758256.897271:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758256.897273:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758256.897277:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758256.897279:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758256.897282:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758256.897284:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758256.897286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758256.897289:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758257.089316:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758257.089323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758257.089330:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758257.089331:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758257.089433:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758257.089454:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758257.089459:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758257.921290:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758257.921300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758257.921303:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758257.921310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758257.921314:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758257.921319:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758257.921322:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758257.921325:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758257.921328:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758258.113305:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758258.113318:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758258.113322:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758258.113328:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758258.113337:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758258.113341:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758258.113345:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758258.113351:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758258.113358:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758258.113360:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758258.113430:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758258.113436:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758258.113440:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758258.945246:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758258.945254:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758258.945256:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758258.945261:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758258.945266:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758258.945269:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758258.945272:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758258.945275:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758258.945277:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758259.137263:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758259.137271:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758259.137277:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758259.137279:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758259.137297:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758259.137299:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758259.137301:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758259.393271:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758259.393278:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787758259.393281:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758259.393283:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758259.393287:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758259.393294:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11314 00000100:00080000:0.0:1787758259.393303:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758259.393305:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758259.393307:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758259.393310:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758259.393312:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758259.393315:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787758259.393317:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787758259.393318:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758259.393320:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758259.393323:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758259.393325:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758259.393327:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758259.393330:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758259.393333:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393334:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758259.393336:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758259.393345:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758259.393346:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393349:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758259.393351:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758259.393355:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393364:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758259.393372:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790d880 x1874588659730048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758259.393380:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758259.393383:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393398:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758259.393401:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758259.393404:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787758259.393405:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758259.393406:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758259.393408:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758259.393410:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11314 00000100:00080000:0.0:1787758259.393413:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758259.393414:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758259.393415:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758259.393418:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758259.393419:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758259.393421:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787758259.393422:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758259.393424:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758259.393425:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758259.393427:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758259.393435:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758259.393437:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758259.393441:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758259.393442:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393444:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758259.393445:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758259.393450:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:0.0:1787758259.393451:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758259.393454:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:0.0:1787758259.393455:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393459:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393459:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758259.393464:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758259.393465:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758259.393468:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393469:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:0.0:1787758259.393470:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758259.393473:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758259.393473:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758259.393477:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393478:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393483:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758259.393486:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659730048:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1787758259.393491:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f100 x1874588659730176/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758259.393493:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758259.393498:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:0.0:1787758259.393502:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787758259.393503:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393506:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393507:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:0.0:1787758259.393510:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393513:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758259.393514:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758259.393520:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758259.393521:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758259.393523:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758259.393525:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758259.393527:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11314 00000100:00080000:0.0:1787758259.393529:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758259.393530:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758259.393532:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758259.393532:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758259.393536:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000020:00000001:0.0:1787758259.393537:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758259.393540:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758259.393543:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758259.393543:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:0.0:1787758259.393547:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758259.393549:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758259.393550:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758259.393553:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758259.393554:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758259.393556:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758259.393558:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758259.393559:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393561:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758259.393562:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758259.393564:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393565:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758259.393569:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758259.393570:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393571:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758259.393572:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758259.393574:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393578:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758259.393582:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790c380 x1874588659730304/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758259.393585:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393589:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758259.393592:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393592:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305987265 00000100:00000001:0.0:1787758259.393596:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758259.393599:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758259.393602:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758259.393604:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787758259.393604:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:0.0:1787758259.393608:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758259.393611:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787758259.393613:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305987265 00000100:00080000:0.0:1787758259.393615:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11314 00000100:00000001:2.0:1787758259.393619:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:0.0:1787758259.393621:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758259.393624:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758259.393625:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758259.393627:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758259.393628:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:0.0:1787758259.393629:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758259.393632:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758259.393633:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758259.393637:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758259.393639:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393640:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758259.393643:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758259.393644:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758259.393649:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758259.393651:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393652:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:0.0:1787758259.393653:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758259.393658:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758259.393658:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 02000000:00000001:0.0:1787758259.393664:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393666:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758259.393666:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:0.0:1787758259.393671:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758259.393672:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:0.0:1787758259.393674:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393676:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758259.393677:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393682:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:0.0:1787758259.393685:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758259.393686:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758259.393687:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758259.393693:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758259.393694:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758259.393695:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758259.393698:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393700:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758259.393703:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393705:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758259.393709:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ce00 x1874588659730432/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758259.393716:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730176:192.168.201.118@tcp:38:kworker.0 00000100:00000001:0.0:1787758259.393718:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758259.393723:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393725:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:0.0:1787758259.393727:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393728:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758259.393732:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393734:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758259.393737:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393744:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758259.393746:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:2.0:1787758259.393751:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758259.393761:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393765:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305986573 00000100:00000001:2.0:1787758259.393767:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305987265 00000100:00000001:2.0:1787758259.393769:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305986573 00000100:00000001:2.0:1787758259.393770:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305987265 00000100:00000001:2.0:1787758259.393772:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758259.393774:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393776:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758259.393777:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758259.393778:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393780:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758259.393781:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758259.393782:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758259.393785:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730304:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758259.393788:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758259.393789:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758259.393790:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393792:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758259.393793:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393797:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758259.393798:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:2.0:1787758259.393800:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758259.393805:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393807:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305986570 00000100:00000001:2.0:1787758259.393809:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305987265 00000100:00000001:2.0:1787758259.393811:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305986570 00000100:00000001:2.0:1787758259.393812:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305987265 00000100:00000001:2.0:1787758259.393814:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758259.393815:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393817:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393818:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758259.393820:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758259.393821:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787758259.393823:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758259.393824:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758259.393824:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758259.393825:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758259.393826:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393827:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758259.393829:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758259.393830:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758259.393832:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758259.393835:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758259.393836:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758259.393837:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758259.393838:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758259.393840:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393842:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758259.393843:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:2.0:1787758259.393846:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758259.393851:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393853:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305986570 00000100:00000001:2.0:1787758259.393854:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305987265 00000100:00000001:2.0:1787758259.393856:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305986570 00000100:00000001:2.0:1787758259.393858:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305987265 00000100:00000001:2.0:1787758259.393859:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758259.393860:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393862:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393863:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758259.393864:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758259.393865:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:2.0:1787758259.393866:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758259.393867:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758259.393868:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.393869:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758259.969253:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758259.969260:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758259.969262:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758259.969267:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758259.969270:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758259.969274:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758259.969277:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758259.969280:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758259.969284:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758260.161267:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758260.161272:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758260.161278:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758260.161279:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758260.161295:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758260.161297:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758260.161299:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758260.993296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758260.993309:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758260.993313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758260.993322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758260.993328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758260.993335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758260.993340:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758260.993345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758260.993351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758261.185291:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758261.185299:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758261.185308:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758261.185311:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758261.185407:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758261.185413:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758261.185418:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758262.017279:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758262.017289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758262.017292:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758262.017297:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758262.017302:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758262.017307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.017310:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758262.017313:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.017316:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758262.209276:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758262.209283:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758262.209289:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758262.209290:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758262.209366:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758262.209370:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758262.209374:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787758262.465472:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1787758262.465517:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787758262.465534:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758262.465548:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758262.465571:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758262.465573:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758262.465576:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787758262.465585:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758262.465589:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758262.465592:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758262.465593:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758262.465595:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787758262.465599:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758262.465602:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758262.465604:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758262.465606:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758262.465607:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:1.0:1787758262.465611:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758262.465615:0:2355:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758262.465617:0:2355:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758262.465618:0:2355:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758262.465619:0:2355:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758262.465674:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758262.465681:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758262.465685:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758262.465700:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758259/real 1787758262] req@ffff8ffb4790ce00 x1874588659730432/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758262.465736:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758262.465740:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758262.465749:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758262.465752:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758262.465755:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.465759:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758262.465762:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758262.465769:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659730432/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758262.465783:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.465788:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758262.465801:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.465804:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758262.465806:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.465809:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758262.465829:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758262.465835:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758262.465842:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758262.465846:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758262.465852:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730432:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758262.465860:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758262.465866:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758259/real 1787758262] req@ffff8ffb4790f100 x1874588659730176/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758262.465880:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758262.465891:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758262.465897:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758262.465899:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758262.465901:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.465904:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758262.465906:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758262.465918:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659730176/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758262.465930:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.465933:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758262.465937:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.465939:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758262.465942:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.465944:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758262.465978:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758262.465983:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758262.465989:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758262.465993:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758262.465998:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730176:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758262.466004:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758262.466010:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758259/real 1787758262] req@ffff8ffb4790c380 x1874588659730304/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758262.466022:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758262.466024:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758262.466029:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758262.466031:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758262.466033:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466036:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758262.466038:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758262.466043:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659730304/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758262.466054:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466057:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758262.466060:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466063:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758262.466065:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466068:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758262.466084:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758262.466088:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758262.466094:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758262.466098:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758262.466102:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730304:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758262.466109:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758262.466115:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758259/real 1787758262] req@ffff8ffb4790d880 x1874588659730048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758262.466127:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758262.466129:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758262.466134:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758262.466137:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758262.466139:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466142:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758262.466144:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758262.466149:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659730048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758262.466159:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466162:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758262.466165:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466168:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758262.466170:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466173:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758262.466175:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758262.466178:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758262.466215:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758262.466219:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758262.466224:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659730048:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758262.466230:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466233:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758262.466235:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758262.466238:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758262.466246:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758262.466249:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758262.466253:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758262.466259:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758262.466261:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758262.466266:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758262.466268:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466271:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758262.466273:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758262.466275:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758262.466277:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758262.466279:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758262.466280:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758262.466283:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758262.466285:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758262.466288:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758262.466290:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466293:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758262.466294:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758262.466296:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758262.466299:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758262.466300:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758262.466302:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758262.466305:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758262.466306:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758262.466309:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758262.466310:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466313:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758262.466314:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758262.466316:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758262.466318:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758262.466320:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758262.466322:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758262.466324:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758262.466325:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758262.466327:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758262.466329:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466332:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758262.466343:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758262.466345:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758262.466348:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758262.466351:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758263.041262:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758263.041270:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758263.041272:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758263.041277:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758263.041282:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758263.041287:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758263.041290:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758263.041294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758263.041300:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758263.233353:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758263.233362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758263.233368:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758263.233370:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758263.233398:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758263.233401:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758263.233405:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758264.065242:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758264.065248:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758264.065250:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758264.065254:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758264.065256:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758264.065259:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.065261:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758264.065264:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.065266:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758264.257292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758264.257302:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758264.257309:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758264.257314:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758264.257317:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758264.257321:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758264.257325:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758264.257328:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758264.257331:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758264.257333:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758264.257359:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758264.257363:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758264.257366:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758264.513372:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758264.513386:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758264.513389:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758264.513394:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758264.513399:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758264.513410:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11319 00000100:00080000:1.0:1787758264.513421:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758264.513425:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758264.513428:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758264.513433:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758264.513436:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758264.513440:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758264.513443:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758264.513446:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758264.513447:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758264.513453:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758264.513457:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758264.513460:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758264.513464:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758264.513468:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513471:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758264.513474:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758264.513485:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758264.513487:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513491:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758264.513494:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758264.513499:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513512:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758264.513523:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659730560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758264.513549:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758264.513555:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513581:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758264.513586:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758264.513591:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758264.513592:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758264.513594:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758264.513598:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758264.513601:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11319 00000100:00080000:1.0:1787758264.513606:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758264.513609:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758264.513611:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758264.513615:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758264.513617:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758264.513619:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758264.513622:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758264.513626:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758264.513628:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758264.513632:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758264.513637:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758264.513639:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758264.513646:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758264.513650:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513652:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758264.513657:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:1.0:1787758264.513659:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758264.513666:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00000001:1.0:1787758264.513667:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513674:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.513675:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1787758264.513681:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758264.513683:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787758264.513686:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.513689:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513691:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758264.513697:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758264.513701:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.513702:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1787758264.513708:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.513710:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758264.513719:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758264.513722:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659730688/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758264.513732:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659730560:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758264.513748:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787758264.513756:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787758264.513758:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758264.513760:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.513766:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513768:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.513774:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:1.0:1787758264.513776:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758264.513783:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758264.513785:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:2.0:1787758264.513787:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758264.513795:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758264.513801:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758264.513807:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11319 00000100:00000001:2.0:1787758264.513812:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787758264.513813:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758264.513821:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758264.513821:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:1.0:1787758264.513826:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758264.513832:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758264.513833:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:1.0:1787758264.513835:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758264.513840:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758264.513845:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758264.513848:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758264.513850:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758264.513854:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758264.513857:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758264.513860:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758264.513865:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758264.513868:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513870:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758264.513873:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758264.513876:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513880:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758264.513887:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758264.513889:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513892:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758264.513894:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758264.513898:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787758264.513903:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758264.513907:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758264.513918:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659730816/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758264.513920:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758264.513953:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758264.513956:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.513959:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758264.513962:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758264.513967:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758264.513969:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758264.513971:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758264.513978:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787758264.513982:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00080000:1.0:1787758264.513984:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11319 00000100:00000001:2.0:1787758264.513991:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758264.513993:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758264.513998:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758264.514000:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758264.514005:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758264.514009:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758264.514010:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758264.514014:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787758264.514018:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305992385 00000020:00000001:1.0:1787758264.514019:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758264.514027:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758264.514031:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758264.514033:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758264.514035:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758264.514039:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758264.514045:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305992385 00000100:00000001:1.0:1787758264.514046:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758264.514054:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758264.514057:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:1.0:1787758264.514060:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758264.514065:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.514068:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758264.514070:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758264.514074:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758264.514076:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758264.514082:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758264.514084:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514086:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758264.514087:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758264.514091:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758264.514095:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758264.514096:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514102:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758264.514106:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758264.514115:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758264/real 1787758264] req@ffff8ffb4a534a80 x1874588659730560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758264.514119:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659730944/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758264.514154:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758264.514157:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787758264.514157:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758264.514163:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758264.514165:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514170:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758264.514177:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758264.514179:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514225:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758264.514228:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758264.514236:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659730560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758264.514247:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514251:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758264.514255:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514257:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758264.514258:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514261:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758264.514263:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758264.514267:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758264.514272:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758264.514275:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758264.514280:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659730560:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758264.514287:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514289:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758264.514291:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758264.514294:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758264.514298:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758264.514302:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758264.514304:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758264.514309:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758264.514311:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758264.514315:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758264.514316:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514319:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514328:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758264.514331:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514333:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758264.514335:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758264.514337:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758264.514339:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758264.514341:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.514344:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758264.514347:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758264.514350:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758264.514354:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730688:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758264.514359:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758264.514361:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758264.514364:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.514366:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758264.514370:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514378:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758264.514380:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787758264.514386:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758264.514405:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758264.514411:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758264.514414:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758264.514415:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514417:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758264.514421:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514426:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305991694 00000100:00000001:2.0:1787758264.514430:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305992385 00000100:00000001:2.0:1787758264.514432:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305991694 00000100:00000001:2.0:1787758264.514435:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305992385 00000100:00000001:2.0:1787758264.514438:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758264.514441:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514444:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758264.514450:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758264/real 1787758264] req@ffff8ffb4a536300 x1874588659730688/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758264.514461:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758264.514463:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758264.514466:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758264.514468:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758264.514470:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514472:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758264.514474:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758264.514478:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659730688/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758264.514488:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514490:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758264.514493:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514495:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758264.514496:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514498:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758264.514518:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758264.514521:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758264.514525:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758264.514528:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758264.514532:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730688:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758264.514537:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758264.514539:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758264.514541:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.514543:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758264.514546:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758264.514549:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758264.514553:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730816:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758264.514557:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758264.514559:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758264.514561:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.514564:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758264.514568:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514575:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758264.514577:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787758264.514583:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758264.514598:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758264.514615:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758264.514618:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758264.514620:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514622:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758264.514625:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514630:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305991691 00000100:00000001:2.0:1787758264.514633:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305992385 00000100:00000001:2.0:1787758264.514636:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305991691 00000100:00000001:2.0:1787758264.514639:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305992385 00000100:00000001:2.0:1787758264.514641:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758264.514644:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514646:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758264.514653:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758264/real 1787758264] req@ffff8ffb4a537b80 x1874588659730816/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758264.514663:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758264.514665:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758264.514668:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758264.514670:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758264.514672:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514674:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758264.514676:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758264.514681:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659730816/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758264.514706:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514709:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758264.514712:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514714:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758264.514716:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514718:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758264.514734:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758264.514738:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758264.514742:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758264.514745:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758264.514749:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730816:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758264.514754:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758264.514756:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758264.514758:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.514760:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758264.514763:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758264.514766:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758264.514770:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758264.514774:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758264.514776:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758264.514778:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758264.514781:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758264.514784:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514790:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758264.514792:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787758264.514796:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758264.514813:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758264.514819:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758264.514822:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758264.514824:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514827:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758264.514831:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514835:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305991691 00000100:00000001:2.0:1787758264.514838:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305992385 00000100:00000001:2.0:1787758264.514841:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305991691 00000100:00000001:2.0:1787758264.514844:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305992385 00000100:00000001:2.0:1787758264.514847:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758264.514850:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.514852:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758264.514858:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758264/real 1787758264] req@ffff8ffb4a537800 x1874588659730944/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758264.514869:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758264.514871:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758264.514874:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758264.514876:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758264.514878:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514880:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758264.514882:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758264.514888:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659730944/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758264.514898:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514901:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758264.514904:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514906:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758264.514907:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514910:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758264.514940:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758264.514944:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758264.514949:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758264.514952:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758264.514957:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659730944:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758264.514961:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514964:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758264.514966:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758264.514968:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758264.514971:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758264.514974:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758264.514976:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758264.514980:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758264.514981:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758264.514983:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758264.514984:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514986:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758264.514987:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758264.514988:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758264.514990:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758264.514991:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758264.514992:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758264.514993:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758264.514994:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758264.514996:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758264.514997:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.514998:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758264.514999:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758264.515000:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758264.515002:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758264.515003:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758264.515004:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758264.515006:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758264.515006:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758264.515007:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758264.515008:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.515010:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758264.515013:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758264.515014:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758264.515016:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758264.515017:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758265.089246:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758265.089255:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758265.089257:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758265.089263:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758265.089266:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758265.089270:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758265.089273:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758265.089276:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758265.089278:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758265.281300:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758265.281306:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758265.281311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758265.281312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758265.281334:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758265.281338:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758265.281341:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758266.113299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758266.113308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758266.113310:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758266.113316:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758266.113320:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758266.113324:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758266.113327:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758266.113329:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758266.113332:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758266.305291:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758266.305297:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758266.305302:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758266.305303:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758266.305355:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758266.305359:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758266.305361:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758267.137260:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758267.137269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758267.137272:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758267.137278:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758267.137283:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758267.137288:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758267.137292:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758267.137307:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758267.137312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758267.329286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758267.329304:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758267.329308:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758267.329313:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758267.329320:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758267.329323:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758267.329328:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758267.329334:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758267.329344:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758267.329347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758267.329379:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758267.329382:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758267.329384:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758268.161282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758268.161290:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758268.161293:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758268.161299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758268.161303:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758268.161307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758268.161310:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758268.161313:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758268.161316:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758268.353271:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758268.353277:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758268.353283:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758268.353285:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758268.353310:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758268.353313:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758268.353318:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758269.185265:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758269.185271:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758269.185272:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758269.185276:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758269.185279:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758269.185283:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.185285:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758269.185287:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.185289:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758269.377319:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758269.377327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758269.377333:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758269.377335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758269.377383:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758269.377389:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758269.377393:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758269.633378:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758269.633393:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758269.633397:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758269.633402:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758269.633408:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758269.633435:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11324 00000100:00080000:1.0:1787758269.633448:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758269.633452:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758269.633454:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758269.633465:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758269.633469:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758269.633472:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758269.633475:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758269.633477:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758269.633479:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758269.633483:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758269.633485:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.633488:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758269.633492:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758269.633495:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633496:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758269.633499:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.633514:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758269.633515:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633518:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758269.633522:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758269.633526:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633539:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758269.633548:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659731072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758269.633558:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758269.633562:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633575:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758269.633578:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758269.633581:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758269.633582:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758269.633583:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758269.633585:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758269.633587:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11324 00000100:00080000:1.0:1787758269.633589:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758269.633590:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758269.633591:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758269.633593:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758269.633594:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758269.633595:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758269.633597:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758269.633599:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758269.633599:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758269.633602:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758269.633604:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.633605:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758269.633610:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758269.633612:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633613:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758269.633617:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758269.633619:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633621:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.633623:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758269.633624:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633626:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758269.633628:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758269.633629:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633633:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758269.633637:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659731200/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758269.633645:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758269.633648:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633651:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758269.633655:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787758269.633657:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787758269.633660:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758269.633665:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787758269.633666:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:1.0:1787758269.633672:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758269.633672:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758269.633677:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758269.633680:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633698:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:2.0:1787758269.633698:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758269.633705:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758269.633710:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11324 02000000:00000001:2.0:1787758269.633712:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758269.633724:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758269.633727:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659731072:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758269.633730:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758269.633733:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758269.633737:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:1.0:1787758269.633743:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758269.633746:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787758269.633747:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:1.0:1787758269.633749:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787758269.633751:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758269.633757:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787758269.633758:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758269.633765:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758269.633766:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787758269.633772:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758269.633773:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758269.633782:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.633785:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758269.633789:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758269.633792:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633794:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758269.633795:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000008:00100000:1.0:1787758269.633797:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758269.633798:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000008:00000001:1.0:1787758269.633808:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.633811:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758269.633813:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.633822:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758269.633823:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633825:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758269.633828:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758269.633831:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633858:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758269.633863:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659731328/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758269.633870:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758269.633890:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633893:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758269.633895:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758269.633898:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758269.633899:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758269.633901:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758269.633910:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758269.633911:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758269.633919:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11324 00000100:00080000:1.0:1787758269.633927:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758269.633929:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758269.633931:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758269.633932:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:1.0:1787758269.633935:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758269.633937:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758269.633939:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758269.633941:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758269.633941:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758269.633948:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758269.633950:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633951:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758269.633963:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758269.633963:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758269.633970:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.633972:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758269.633975:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758269.633976:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.633977:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633986:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758269.633988:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758269.633990:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4305997505 00000008:00000001:1.0:1787758269.633991:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.633996:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758269.634001:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758269.634002:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.634003:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758269.634005:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758269.634008:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634009:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758269.634013:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634016:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4305997505 00000100:00100000:1.0:1787758269.634021:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659731456/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758269.634024:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758269.634036:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758269.634037:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758269.634039:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634044:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758269.634047:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634050:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758269.634059:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758269/real 1787758269] req@ffff8ffb4a534700 x1874588659731072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758269.634076:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758269.634079:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758269.634087:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758269.634102:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758269.634104:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634107:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758269.634109:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758269.634113:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659731072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758269.634119:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634122:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758269.634124:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634126:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758269.634127:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634129:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758269.634130:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758269.634140:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758269.634144:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758269.634145:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758269.634148:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659731072:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758269.634152:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758269.634154:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758269.634155:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758269.634157:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758269.634159:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758269.634161:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758269.634163:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731200:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758269.634166:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758269.634167:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758269.634169:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758269.634171:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758269.634174:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634178:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758269.634180:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787758269.634236:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758269.634257:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758269.634262:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758269.634264:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758269.634265:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634267:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758269.634269:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634272:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305996814 00000100:00000001:2.0:1787758269.634275:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305997505 00000100:00000001:2.0:1787758269.634277:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305996814 00000100:00000001:2.0:1787758269.634279:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305997505 00000100:00000001:2.0:1787758269.634281:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758269.634282:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634284:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758269.634294:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758269/real 1787758269] req@ffff8ffb4a535c00 x1874588659731200/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758269.634300:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758269.634301:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758269.634303:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758269.634304:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758269.634305:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634307:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758269.634308:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758269.634311:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659731200/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758269.634316:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634335:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758269.634339:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634341:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758269.634343:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634345:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758269.634372:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758269.634375:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758269.634379:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758269.634382:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758269.634387:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731200:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758269.634391:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634394:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758269.634396:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758269.634399:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758269.634404:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758269.634407:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758269.634410:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758269.634416:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758269.634418:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758269.634433:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758269.634434:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634436:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758269.634438:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758269.634439:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758269.634442:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758269.634443:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758269.634444:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758269.634446:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758269.634448:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758269.634450:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758269.634451:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634470:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634478:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758269.634480:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634483:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758269.634485:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758269.634486:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758269.634488:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758269.634490:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758269.634494:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758269.634496:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758269.634498:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758269.634502:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731328:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758269.634505:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758269.634508:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758269.634509:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758269.634511:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758269.634515:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634522:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758269.634524:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787758269.634528:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758269.634542:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758269.634547:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758269.634550:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758269.634551:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634553:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758269.634557:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634560:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305996811 00000100:00000001:2.0:1787758269.634563:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305997505 00000100:00000001:2.0:1787758269.634566:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305996811 00000100:00000001:2.0:1787758269.634568:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305997505 00000100:00000001:2.0:1787758269.634571:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758269.634573:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634575:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758269.634580:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758269/real 1787758269] req@ffff8ffb4a537800 x1874588659731328/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758269.634588:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758269.634589:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758269.634592:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758269.634610:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758269.634612:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634613:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758269.634614:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758269.634617:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659731328/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758269.634623:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634624:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758269.634626:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634627:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758269.634628:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634629:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758269.634638:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758269.634640:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758269.634642:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758269.634644:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758269.634646:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731328:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758269.634649:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758269.634650:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758269.634651:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758269.634653:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758269.634654:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758269.634656:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758269.634658:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758269.634660:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758269.634661:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758269.634662:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758269.634664:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758269.634665:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634669:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758269.634670:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787758269.634673:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758269.634680:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758269.634683:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758269.634685:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758269.634686:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634687:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758269.634689:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634691:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4305996811 00000100:00000001:2.0:1787758269.634693:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4305997505 00000100:00000001:2.0:1787758269.634695:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4305996811 00000100:00000001:2.0:1787758269.634696:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4305997505 00000100:00000001:2.0:1787758269.634698:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758269.634699:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634701:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758269.634712:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758269/real 1787758269] req@ffff8ffb4a537b80 x1874588659731456/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758269.634718:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758269.634718:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758269.634720:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758269.634721:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758269.634722:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634723:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758269.634724:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758269.634727:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659731456/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758269.634732:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634733:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758269.634735:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634736:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758269.634737:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634739:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758269.634748:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758269.634750:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758269.634753:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758269.634754:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758269.634757:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731456:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758269.634760:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634761:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758269.634762:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758269.634763:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758269.634764:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758269.634765:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758269.634766:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758269.634778:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758269.634779:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758269.634780:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758269.634781:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634783:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758269.634783:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758269.634784:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758269.634786:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758269.634787:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758269.634787:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758269.634789:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758269.634790:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758269.634791:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758269.634792:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634793:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758269.634795:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758269.634796:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758269.634797:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758269.634799:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758270.209262:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758270.209275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758270.209277:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758270.209282:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758270.209285:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758270.209288:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758270.209290:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758270.209292:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758270.209294:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758270.401350:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758270.401362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758270.401372:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758270.401375:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758270.401467:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758270.401476:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758270.401484:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758271.233340:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758271.233349:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758271.233352:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758271.233357:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758271.233361:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758271.233365:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758271.233368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758271.233371:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758271.233374:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758271.425332:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758271.425340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758271.425346:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758271.425347:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758271.425376:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758271.425379:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758271.425383:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758272.257307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758272.257316:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758272.257319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758272.257325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758272.257330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758272.257335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758272.257338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758272.257341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758272.257344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758272.449325:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758272.449335:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758272.449342:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758272.449344:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758272.449371:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758272.449375:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758272.449379:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758273.281257:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758273.281275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758273.281278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758273.281283:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758273.281286:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758273.281290:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758273.281292:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758273.281294:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758273.281296:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758273.473379:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758273.473390:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758273.473400:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758273.473402:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758273.473459:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758273.473473:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758273.473477:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758274.305306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758274.305314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758274.305317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758274.305324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758274.305328:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758274.305333:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758274.305336:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758274.305340:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758274.305343:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758274.497315:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758274.497323:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758274.497329:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758274.497334:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758274.497337:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758274.497340:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758274.497344:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758274.497347:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758274.497350:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758274.497352:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758274.497473:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758274.497479:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758274.497484:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758274.753338:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758274.753351:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758274.753354:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758274.753359:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758274.753370:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758274.753382:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11329 00000100:00080000:2.0:1787758274.753399:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758274.753405:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758274.753410:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758274.753441:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758274.753459:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758274.753467:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758274.753473:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758274.753477:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758274.753481:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758274.753489:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758274.753492:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758274.753495:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758274.753499:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758274.753503:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.753505:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758274.753508:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758274.753519:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758274.753521:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.753525:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758274.753528:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758274.753533:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.753546:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758274.753576:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659731584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758274.753590:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758274.753595:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.753621:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758274.753627:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758274.753635:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758274.753639:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758274.753642:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758274.753647:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758274.753652:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11329 00000100:00080000:2.0:1787758274.753663:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758274.753666:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758274.753670:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758274.753671:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:2.0:1787758274.753675:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758274.753698:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758274.753700:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787758274.753703:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758274.753708:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:2.0:1787758274.753709:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758274.753716:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758274.753718:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758274.753719:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758274.753727:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758274.753728:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758274.753738:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.753740:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758274.753761:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758274.753771:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758274.753771:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758274.753785:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758274.753791:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758274.753795:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659731584:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758274.753797:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758274.753805:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758274.753806:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:2.0:1787758274.753813:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.753814:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787758274.753817:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.753820:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.753826:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758274.753826:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758274.753832:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.753837:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758274.753838:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758274.753839:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758274.753845:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.753856:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.753862:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:2.0:1787758274.753865:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659731712/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758274.753867:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:1.0:1787758274.753878:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758274.753893:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758274.753897:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.753902:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758274.753905:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758274.753910:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758274.753912:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758274.753915:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758274.753918:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758274.753924:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11329 00000800:00000100:1.0:1787758274.753932:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787758274.753936:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758274.753941:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758274.753943:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758274.753947:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:2.0:1787758274.753948:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758274.753954:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758274.753957:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000020:00000001:2.0:1787758274.753957:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758274.753963:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758274.753964:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758274.753971:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758274.753972:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758274.753978:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758274.753982:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758274.753984:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758274.753992:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.753995:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306002625 02000000:00000001:2.0:1787758274.753996:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758274.754001:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758274.754006:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.754007:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758274.754010:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758274.754014:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00000001:2.0:1787758274.754015:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.754019:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758274.754028:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758274.754029:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306002625 02000000:00000001:2.0:1787758274.754030:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754043:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787758274.754047:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758274.754073:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758274.754075:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758274.754084:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754088:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:2.0:1787758274.754094:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659731840/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758274.754099:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754109:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758274.754117:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758274.754123:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758274.754125:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758274/real 1787758274] req@ffff8ffb47a38700 x1874588659731584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758274.754128:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758274.754137:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758274.754150:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758274.754157:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758274.754162:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000400:00000001:1.0:1787758274.754163:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:2.0:1787758274.754167:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000400:00000001:1.0:1787758274.754169:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:2.0:1787758274.754177:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11329 00000100:00080000:2.0:1787758274.754201:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758274.754204:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758274.754206:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758274.754211:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758274.754213:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758274.754216:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758274.754228:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758274.754237:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758274.754242:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754246:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758274.754250:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758274.754258:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659731584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787758274.754273:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758274.754275:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758274.754280:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758274.754283:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758274.754285:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758274.754292:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754300:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:2.0:1787758274.754300:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758274.754306:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758274.754308:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754311:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:2.0:1787758274.754313:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758274.754317:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:2.0:1787758274.754324:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758274.754326:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:2.0:1787758274.754335:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758274.754340:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758274.754342:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758274.754350:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758274.754351:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000008:00000001:2.0:1787758274.754364:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758274.754370:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659731584:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758274.754374:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754383:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754390:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787758274.754391:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758274.754396:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758274.754398:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.754404:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:2.0:1787758274.754406:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758274.754413:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:2.0:1787758274.754414:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000020:00000001:1.0:1787758274.754420:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:2.0:1787758274.754423:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758274.754425:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758274.754458:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758274.754462:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758274.754464:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754467:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758274.754472:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754475:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758274.754484:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659731968/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758274.754486:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758274.754492:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754499:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758274.754503:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758274.754506:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758274.754507:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758274.754513:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754515:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758274.754521:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754522:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.754531:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758274.754536:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758274.754540:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758274.754549:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731712:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758274.754557:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758274.754561:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758274.754564:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.754568:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758274.754574:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754585:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758274.754588:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:1.0:1787758274.754595:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758274.754616:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758274.754623:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758274.754626:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758274.754628:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754630:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758274.754633:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754637:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306001934 00000100:00000001:1.0:1787758274.754641:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306002625 00000100:00000001:1.0:1787758274.754644:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306001934 00000100:00000001:1.0:1787758274.754647:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306002625 00000100:00000001:1.0:1787758274.754649:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758274.754651:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754653:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758274.754658:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758274/real 1787758274] req@ffff8ffb47a3a300 x1874588659731712/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758274.754667:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758274.754669:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758274.754671:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758274.754674:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758274.754677:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754680:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758274.754682:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758274.754689:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659731712/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758274.754701:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754705:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758274.754709:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754712:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758274.754715:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754718:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758274.754737:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758274.754762:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758274.754770:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758274.754773:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758274.754777:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731712:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758274.754782:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758274.754784:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758274.754786:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.754788:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758274.754790:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758274.754793:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758274.754796:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731840:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758274.754800:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758274.754803:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758274.754805:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.754807:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758274.754811:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754818:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758274.754819:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787758274.754824:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758274.754841:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758274.754846:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758274.754849:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758274.754851:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754853:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758274.754856:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754860:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306001931 00000100:00000001:1.0:1787758274.754863:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306002625 00000100:00000001:1.0:1787758274.754865:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306001931 00000100:00000001:1.0:1787758274.754868:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306002625 00000100:00000001:1.0:1787758274.754870:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758274.754872:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754874:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758274.754879:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758274/real 1787758274] req@ffff8ffb47a39180 x1874588659731840/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758274.754887:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758274.754888:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758274.754891:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758274.754893:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758274.754894:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754896:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758274.754898:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758274.754902:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659731840/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758274.754910:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754912:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758274.754914:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754916:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758274.754917:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754919:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758274.754930:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758274.754933:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758274.754937:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758274.754939:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758274.754943:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731840:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758274.754947:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754949:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758274.754950:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758274.754952:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758274.754954:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758274.754956:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758274.754957:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758274.754960:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758274.754961:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758274.754964:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758274.754966:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754967:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758274.754969:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758274.754970:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758274.754973:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758274.754974:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758274.754975:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758274.754977:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758274.754978:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758274.754980:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758274.754981:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754983:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.754987:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758274.754989:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.754991:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758274.754992:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758274.754993:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758274.754995:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758274.754996:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.754998:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758274.755000:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758274.755002:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758274.755005:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731968:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758274.755011:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758274.755014:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758274.755017:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758274.755020:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758274.755024:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.755032:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758274.755035:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787758274.755041:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758274.755077:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758274.755085:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758274.755089:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758274.755092:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.755096:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758274.755100:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.755106:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306001931 00000100:00000001:1.0:1787758274.755111:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306002625 00000100:00000001:1.0:1787758274.755115:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306001931 00000100:00000001:1.0:1787758274.755119:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306002625 00000100:00000001:1.0:1787758274.755123:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758274.755126:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.755128:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758274.755134:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758274/real 1787758274] req@ffff8ffb47a3b480 x1874588659731968/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758274.755144:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758274.755146:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758274.755149:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758274.755151:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758274.755153:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.755155:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758274.755157:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758274.755161:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659731968/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758274.755169:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.755171:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758274.755175:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.755177:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758274.755178:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.755181:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758274.755193:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758274.755256:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758274.755263:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758274.755267:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758274.755271:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659731968:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758274.755278:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.755281:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758274.755283:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758274.755287:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758274.755293:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758274.755296:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758274.755297:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758274.755303:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758274.755305:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758274.755308:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758274.755310:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.755312:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758274.755317:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758274.755320:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758274.755324:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758274.755326:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758275.329293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758275.329302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758275.329306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758275.329312:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758275.329316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758275.329320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758275.329324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758275.329327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758275.329331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758275.521275:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758275.521281:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758275.521286:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758275.521287:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758275.521306:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758275.521309:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758275.521311:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758276.353300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758276.353310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758276.353313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758276.353319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758276.353324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758276.353329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758276.353333:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758276.353336:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758276.353339:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758276.545279:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758276.545286:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758276.545292:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758276.545293:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758276.545326:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758276.545328:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758276.545331:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758277.377273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758277.377279:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758277.377282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758277.377286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758277.377289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758277.377293:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758277.377295:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758277.377297:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758277.377299:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758277.569278:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758277.569288:0:178246:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758277.569291:0:178246:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758277.569294:0:178246:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758277.569300:0:178246:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758277.569303:0:178246:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758277.569305:0:178246:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758277.569307:0:178246:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758277.569311:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758277.569312:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758277.569345:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758277.569350:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758277.569353:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758278.401355:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758278.401366:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758278.401370:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758278.401378:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758278.401383:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758278.401405:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758278.401409:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758278.401413:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758278.401417:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758278.593343:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758278.593352:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758278.593360:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758278.593362:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758278.593458:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758278.593472:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758278.593478:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758279.425293:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758279.425302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758279.425304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758279.425311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758279.425315:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758279.425319:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.425322:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758279.425325:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.425327:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758279.617270:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758279.617277:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758279.617284:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758279.617286:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758279.617314:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758279.617317:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758279.617321:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758279.873344:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758279.873362:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758279.873367:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758279.873373:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758279.873381:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758279.873393:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11334 00000100:00080000:1.0:1787758279.873410:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758279.873417:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758279.873421:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758279.873430:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758279.873435:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758279.873440:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758279.873445:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758279.873450:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758279.873453:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758279.873462:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758279.873468:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758279.873474:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758279.873480:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758279.873486:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.873491:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758279.873506:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758279.873523:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758279.873526:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.873531:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758279.873534:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758279.873539:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.873554:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758279.873566:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659732096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758279.873580:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758279.873597:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.873622:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758279.873627:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758279.873633:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758279.873635:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758279.873637:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758279.873641:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758279.873645:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11334 00000100:00080000:1.0:1787758279.873672:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758279.873678:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758279.873683:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758279.873689:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758279.873692:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758279.873706:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758279.873711:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758279.873716:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758279.873717:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758279.873723:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758279.873728:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:1.0:1787758279.873739:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758279.873739:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758279.873745:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758279.873752:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.873753:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758279.873766:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758279.873766:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758279.873775:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758279.873779:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758279.873782:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:2.0:1787758279.873790:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659732096:192.168.201.118@tcp:250:kworker.0 02000000:00000001:1.0:1787758279.873793:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.873803:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758279.873806:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00100000:1.0:1787758279.873812:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:2.0:1787758279.873815:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758279.873823:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:1.0:1787758279.873824:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.873831:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758279.873833:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.873841:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758279.873844:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758279.873849:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.873852:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758279.873855:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758279.873859:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.873866:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758279.873871:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.873872:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:2.0:1787758279.873885:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:1.0:1787758279.873889:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659732224/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758279.873904:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758279.873907:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.873914:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758279.873918:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758279.873922:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758279.873926:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758279.873930:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758279.873934:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758279.873937:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11334 00000100:00080000:1.0:1787758279.873945:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758279.873951:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000800:00000100:2.0:1787758279.873956:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758279.873957:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758279.873966:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758279.873968:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758279.873971:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758279.873975:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758279.873978:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758279.873979:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758279.873993:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758279.873995:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758279.873997:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758279.874002:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758279.874006:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787758279.874008:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758279.874014:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758279.874015:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874025:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758279.874026:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758279.874035:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758279.874040:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874040:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.874052:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874055:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306007745 02000000:00000001:1.0:1787758279.874059:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758279.874064:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.874066:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758279.874068:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758279.874073:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:1.0:1787758279.874074:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874087:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306007745 00000100:00000001:1.0:1787758279.874093:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874100:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00100000:1.0:1787758279.874112:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659732352/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758279.874115:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758279.874128:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874138:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758279.874165:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758279/real 1787758279] req@ffff8ffb4a536300 x1874588659732096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758279.874233:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758279.874237:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758279.874247:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758279.874248:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758279.874256:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874259:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758279.874262:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874268:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:1.0:1787758279.874270:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787758279.874280:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:1.0:1787758279.874282:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758279.874285:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758279.874287:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758279.874293:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758279.874300:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659732096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758279.874301:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758279.874313:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11334 00000100:00080000:1.0:1787758279.874326:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758279.874332:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758279.874335:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758279.874340:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758279.874348:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:2.0:1787758279.874350:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1787758279.874356:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758279.874358:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:1.0:1787758279.874370:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758279.874371:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758279.874385:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758279.874386:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758279.874393:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758279.874395:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758279.874404:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00080000:1.0:1787758279.874412:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758279.874414:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758279.874424:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758279.874428:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:1.0:1787758279.874433:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00080000:2.0:1787758279.874444:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:1.0:1787758279.874445:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758279.874458:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:1.0:1787758279.874459:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.874472:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:2.0:1787758279.874475:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659732096:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1787758279.874480:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758279.874485:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:1.0:1787758279.874490:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874492:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758279.874498:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:1.0:1787758279.874502:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.874504:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758279.874512:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758279.874517:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:1.0:1787758279.874518:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000020:00000001:2.0:1787758279.874521:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758279.874524:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758279.874529:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758279.874531:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758279.874532:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758279.874537:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 02000000:00000001:1.0:1787758279.874540:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874544:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758279.874549:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758279.874551:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874553:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.874570:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00100000:1.0:1787758279.874571:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659732480/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758279.874576:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874587:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758279.874592:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758279.874596:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758279.874600:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758279.874602:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874609:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758279.874613:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874616:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.874625:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758279.874630:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758279.874634:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758279.874644:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732224:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758279.874677:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758279.874682:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758279.874685:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.874688:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758279.874694:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874706:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758279.874709:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787758279.874716:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758279.874746:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758279.874756:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758279.874761:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758279.874763:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874766:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758279.874772:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874779:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306007054 00000100:00000001:2.0:1787758279.874784:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306007745 00000100:00000001:2.0:1787758279.874788:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306007054 00000100:00000001:2.0:1787758279.874792:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306007745 00000100:00000001:2.0:1787758279.874796:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758279.874800:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874804:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758279.874812:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758279/real 1787758279] req@ffff8ffb4a534a80 x1874588659732224/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758279.874829:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758279.874832:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758279.874837:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758279.874840:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758279.874843:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.874847:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758279.874849:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758279.874857:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659732224/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758279.874871:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.874875:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758279.874880:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.874883:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758279.874885:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.874889:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758279.874924:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758279.874931:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758279.874938:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758279.874943:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758279.874949:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732224:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758279.874955:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758279.874958:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758279.874960:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.874963:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758279.874965:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758279.874968:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758279.874972:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732352:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758279.874976:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758279.874979:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758279.874981:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.874984:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758279.874988:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.874997:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758279.874998:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787758279.875003:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758279.875021:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758279.875027:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758279.875031:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758279.875033:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875035:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758279.875039:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875043:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306007051 00000100:00000001:2.0:1787758279.875047:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306007745 00000100:00000001:2.0:1787758279.875049:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306007051 00000100:00000001:2.0:1787758279.875052:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306007745 00000100:00000001:2.0:1787758279.875055:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758279.875058:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875060:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758279.875066:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758279/real 1787758279] req@ffff8ffb4a537b80 x1874588659732352/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758279.875077:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758279.875079:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758279.875081:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758279.875084:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758279.875086:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875089:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758279.875091:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758279.875099:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659732352/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758279.875113:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875118:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758279.875123:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875126:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758279.875129:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875132:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758279.875152:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758279.875158:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758279.875177:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758279.875219:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758279.875226:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732352:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758279.875232:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875235:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758279.875237:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758279.875239:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758279.875243:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758279.875245:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758279.875247:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758279.875251:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758279.875252:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758279.875255:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758279.875258:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875260:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758279.875261:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758279.875263:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758279.875265:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758279.875266:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758279.875268:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758279.875270:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758279.875271:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758279.875273:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758279.875275:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875277:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875282:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758279.875284:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875286:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758279.875288:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758279.875290:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758279.875291:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758279.875293:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.875296:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758279.875298:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758279.875301:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758279.875305:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758279.875309:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758279.875312:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758279.875314:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758279.875316:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758279.875320:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875328:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758279.875330:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787758279.875335:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758279.875352:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758279.875358:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758279.875361:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758279.875363:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875365:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758279.875369:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875375:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306007052 00000100:00000001:2.0:1787758279.875381:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306007745 00000100:00000001:2.0:1787758279.875385:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306007052 00000100:00000001:2.0:1787758279.875389:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306007745 00000100:00000001:2.0:1787758279.875393:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758279.875397:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875400:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758279.875409:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758279/real 1787758279] req@ffff8ffb4a537800 x1874588659732480/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758279.875425:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758279.875428:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758279.875434:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758279.875437:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758279.875440:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875444:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758279.875447:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758279.875455:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659732480/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758279.875470:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875475:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758279.875479:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875482:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758279.875483:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875486:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758279.875499:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758279.875503:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758279.875507:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758279.875510:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758279.875515:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732480:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758279.875520:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875523:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758279.875525:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758279.875527:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758279.875530:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758279.875532:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758279.875534:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758279.875537:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758279.875538:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758279.875541:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758279.875542:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875545:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758279.875549:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758279.875551:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758279.875554:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758279.875556:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758280.449303:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758280.449312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758280.449315:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758280.449322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758280.449327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758280.449331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758280.449353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758280.449367:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758280.449371:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758280.641319:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758280.641327:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758280.641337:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758280.641340:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758280.641383:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758280.641386:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758280.641390:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758281.473266:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758281.473272:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758281.473273:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758281.473278:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758281.473281:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758281.473283:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758281.473285:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758281.473288:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758281.473289:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758281.665254:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758281.665260:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758281.665264:0:178246:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758281.665265:0:178246:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758281.665304:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758281.665309:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758281.665313:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758282.497318:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758282.497327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758282.497329:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758282.497335:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758282.497340:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758282.497344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758282.497347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758282.497351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758282.497355:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758282.689335:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758282.689344:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758282.689352:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758282.689355:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758282.689479:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758282.689485:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758282.689490:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758283.521302:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758283.521312:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758283.521316:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758283.521324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758283.521329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758283.521335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758283.521340:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758283.521344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758283.521349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758283.713287:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758283.713294:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758283.713302:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758283.713304:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758283.713327:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758283.713330:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758283.713333:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758284.545320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758284.545332:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758284.545336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758284.545345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758284.545351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758284.545358:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.545362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758284.545367:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.545372:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758284.737357:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758284.737369:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758284.737382:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758284.737389:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758284.737395:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758284.737402:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758284.737408:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758284.737413:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758284.737419:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758284.737424:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758284.737454:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758284.737457:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758284.737461:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758284.993362:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758284.993380:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758284.993384:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758284.993390:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758284.993397:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758284.993409:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11339 00000100:00080000:1.0:1787758284.993428:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758284.993432:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758284.993436:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758284.993442:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758284.993446:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758284.993451:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758284.993454:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758284.993457:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758284.993459:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758284.993466:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758284.993470:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758284.993473:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758284.993478:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758284.993483:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993486:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758284.993489:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758284.993509:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758284.993512:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993517:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758284.993521:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758284.993529:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993548:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758284.993595:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659732608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758284.993612:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758284.993618:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993643:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758284.993648:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758284.993652:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758284.993654:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758284.993657:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758284.993661:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758284.993664:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11339 00000100:00080000:1.0:1787758284.993670:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758284.993673:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758284.993674:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758284.993677:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758284.993679:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758284.993682:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758284.993685:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758284.993688:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758284.993691:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758284.993695:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758284.993698:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758284.993700:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758284.993708:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758284.993711:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993714:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758284.993722:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758284.993726:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993730:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758284.993740:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758284.993743:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993746:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758284.993748:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758284.993751:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993759:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758284.993767:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659732736/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758284.993770:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758284.993785:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758284.993786:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758284.993796:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758284.993797:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.993808:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758284.993812:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.993813:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758284.993834:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787758284.993836:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:1.0:1787758284.993848:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:2.0:1787758284.993849:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758284.993852:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758284.993863:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:2.0:1787758284.993864:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.993883:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:2.0:1787758284.993893:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659732608:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787758284.993895:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11339 00000100:00080000:1.0:1787758284.993912:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758284.993916:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1787758284.993918:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758284.993926:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787758284.993930:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:1.0:1787758284.993931:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758284.993941:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:2.0:1787758284.993943:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758284.993950:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787758284.993951:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:1.0:1787758284.993961:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758284.993965:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758284.993966:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787758284.993970:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758284.993981:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758284.993989:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758284.993992:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758284.993997:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758284.994001:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994002:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758284.994004:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758284.994011:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758284.994017:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.994019:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994024:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 02000000:00000001:1.0:1787758284.994028:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758284.994035:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994036:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758284.994039:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758284.994044:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758284.994049:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.994058:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758284.994070:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659732864/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758284.994081:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758284.994085:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.994088:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758284.994093:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758284.994098:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758284.994109:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000800:00000100:2.0:1787758284.994111:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787758284.994114:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758284.994122:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758284.994129:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11339 00000100:00000001:2.0:1787758284.994134:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787758284.994136:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758284.994142:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758284.994143:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758284.994148:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.994152:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758284.994154:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758284.994162:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758284.994165:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758284.994171:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306012865 00000020:00000001:1.0:1787758284.994172:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758284.994233:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758284.994241:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758284.994253:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758284.994263:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758284.994269:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758284.994270:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306012866 00000100:00000001:2.0:1787758284.994284:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787758284.994287:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758284.994295:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758284.994297:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758284.994302:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758284.994304:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758284.994314:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758284.994315:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.994327:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758284.994329:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:1.0:1787758284.994347:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758284.994362:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758284.994364:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758284/real 1787758284] req@ffff8ffb4a535c00 x1874588659732608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758284.994378:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758284.994396:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758284.994399:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758284.994409:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758284.994416:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000400:00000001:2.0:1787758284.994421:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:1.0:1787758284.994422:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787758284.994430:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758284.994435:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994444:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00100000:1.0:1787758284.994448:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659732992/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758284.994454:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758284.994461:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758284.994466:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758284.994477:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994477:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758284.994488:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758284.994489:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758284.994521:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659732608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758284.994535:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994541:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758284.994546:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994549:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758284.994551:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994554:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758284.994557:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758284.994562:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758284.994592:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758284.994597:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758284.994603:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659732608:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758284.994610:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994613:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758284.994615:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758284.994618:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758284.994624:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758284.994628:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758284.994630:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758284.994636:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758284.994639:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758284.994643:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758284.994645:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994648:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994658:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758284.994661:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994666:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758284.994669:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758284.994672:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758284.994675:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758284.994678:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758284.994682:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758284.994686:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758284.994691:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758284.994697:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732736:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758284.994705:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758284.994709:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758284.994712:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758284.994716:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758284.994722:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994735:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758284.994738:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:2.0:1787758284.994746:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758284.994774:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758284.994781:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758284.994784:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758284.994787:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994789:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758284.994793:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994798:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306012174 00000100:00000001:2.0:1787758284.994802:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306012865 00000100:00000001:2.0:1787758284.994805:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306012174 00000100:00000001:2.0:1787758284.994808:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306012866 00000100:00000001:2.0:1787758284.994811:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758284.994813:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994816:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758284.994822:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758284/real 1787758284] req@ffff8ffb4a534700 x1874588659732736/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758284.994833:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758284.994834:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758284.994838:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758284.994840:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758284.994842:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994844:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758284.994846:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758284.994851:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659732736/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758284.994860:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994863:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758284.994866:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994868:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758284.994869:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.994871:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758284.994886:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758284.994889:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758284.994893:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758284.994896:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758284.994901:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732736:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758284.994906:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758284.994908:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758284.994911:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758284.994914:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758284.994916:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758284.994919:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758284.994922:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732864:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758284.994927:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758284.994930:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758284.994932:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758284.994934:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758284.994938:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994944:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758284.994945:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787758284.994950:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758284.994963:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758284.994968:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758284.994970:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758284.994972:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994974:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758284.994977:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994981:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306012171 00000100:00000001:2.0:1787758284.994985:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306012865 00000100:00000001:2.0:1787758284.994987:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306012171 00000100:00000001:2.0:1787758284.994990:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306012866 00000100:00000001:2.0:1787758284.994993:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758284.994995:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.994997:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758284.995002:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758284/real 1787758284] req@ffff8ffb4a537800 x1874588659732864/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758284.995019:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758284.995021:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758284.995024:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758284.995026:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758284.995027:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995029:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758284.995031:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758284.995035:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659732864/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758284.995043:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995046:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758284.995048:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995050:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758284.995052:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995054:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758284.995066:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758284.995069:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758284.995072:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758284.995075:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758284.995080:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732864:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758284.995084:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758284.995086:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758284.995087:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758284.995091:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758284.995092:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758284.995095:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758284.995098:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758284.995102:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758284.995104:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758284.995107:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758284.995109:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758284.995114:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.995119:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758284.995121:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787758284.995125:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758284.995136:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758284.995141:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758284.995143:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758284.995145:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.995147:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758284.995150:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.995153:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306012171 00000100:00000001:2.0:1787758284.995156:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306012865 00000100:00000001:2.0:1787758284.995159:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306012171 00000100:00000001:2.0:1787758284.995162:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306012866 00000100:00000001:2.0:1787758284.995165:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758284.995167:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.995169:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758284.995174:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758284/real 1787758284] req@ffff8ffb4a537b80 x1874588659732992/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758284.995182:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758284.995184:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758284.995223:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758284.995225:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758284.995227:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995229:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758284.995231:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758284.995237:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659732992/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758284.995250:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995255:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758284.995260:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995263:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758284.995266:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995269:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758284.995290:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758284.995295:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758284.995303:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758284.995308:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758284.995314:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659732992:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758284.995322:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995326:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758284.995328:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758284.995331:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758284.995335:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758284.995338:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758284.995340:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758284.995345:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758284.995347:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758284.995351:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758284.995354:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995357:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758284.995360:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758284.995362:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758284.995365:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758284.995367:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758284.995370:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758284.995374:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758284.995376:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758284.995380:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758284.995383:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995386:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758284.995388:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758284.995390:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758284.995394:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758284.995396:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758284.995398:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758284.995402:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758284.995405:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758284.995408:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758284.995411:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995415:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758284.995422:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758284.995426:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758284.995430:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758284.995433:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758285.569263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758285.569269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758285.569270:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758285.569274:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758285.569277:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758285.569280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758285.569282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758285.569284:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758285.569286:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758285.761301:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758285.761307:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758285.761312:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758285.761313:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758285.761360:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758285.761363:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758285.761368:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758286.593313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758286.593320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758286.593322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758286.593327:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758286.593331:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758286.593334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758286.593337:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758286.593339:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758286.593341:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758286.785303:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758286.785309:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758286.785315:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758286.785316:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758286.785371:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758286.785374:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758286.785378:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758287.617298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758287.617310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758287.617314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758287.617322:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758287.617329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758287.617335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758287.617339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758287.617344:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758287.617349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758287.809367:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758287.809379:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758287.809382:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758287.809386:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758287.809392:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758287.809395:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758287.809397:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758287.809399:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758287.809404:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758287.809405:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758287.809430:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758287.809433:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758287.809436:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758288.641346:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758288.641354:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758288.641357:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758288.641363:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758288.641367:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758288.641371:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758288.641373:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758288.641376:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758288.641379:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758288.833318:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758288.833327:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758288.833335:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758288.833336:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758288.833361:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758288.833365:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758288.833368:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758289.665248:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758289.665255:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758289.665257:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758289.665261:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758289.665263:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758289.665266:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758289.665268:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758289.665270:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758289.665271:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758289.857266:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758289.857272:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758289.857277:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758289.857278:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758289.857334:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758289.857338:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758289.857341:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758290.113340:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758290.113351:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787758290.113354:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758290.113358:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758290.113362:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758290.113369:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11344 00000100:00080000:0.0:1787758290.113381:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758290.113385:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758290.113387:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758290.113392:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758290.113394:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758290.113397:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787758290.113399:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787758290.113401:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758290.113403:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758290.113407:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758290.113409:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758290.113412:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758290.113415:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758290.113419:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113421:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758290.113423:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758290.113434:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758290.113436:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113440:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758290.113442:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758290.113447:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113459:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758290.113469:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790d500 x1874588659733120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758290.113505:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758290.113511:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113531:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758290.113536:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758290.113541:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787758290.113544:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758290.113546:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758290.113550:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758290.113554:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11344 00000100:00080000:0.0:1787758290.113559:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758290.113563:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758290.113566:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758290.113571:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758290.113573:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758290.113576:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787758290.113580:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758290.113583:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758290.113586:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758290.113591:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:0.0:1787758290.113593:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758290.113599:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758290.113601:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.113604:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:0.0:1787758290.113609:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758290.113610:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758290.113615:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.113620:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758290.113621:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758290.113624:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758290.113629:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.113630:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113639:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:2.0:1787758290.113643:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659733120:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1787758290.113647:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758290.113652:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:0.0:1787758290.113653:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113657:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.113658:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758290.113662:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758290.113664:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758290.113665:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:0.0:1787758290.113668:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113671:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758290.113673:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758290.113674:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758290.113676:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113686:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.113693:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00100000:0.0:1787758290.113694:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790e300 x1874588659733248/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758290.113698:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133078272 : -123165576473344 : ffff8ffb4790d500) 00000100:00000001:2.0:1787758290.113710:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758290.113712:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758290.113717:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113721:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758290.113724:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758290.113727:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758290.113729:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758290.113731:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758290.113734:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758290.113737:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11344 00000100:00080000:0.0:1787758290.113742:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758290.113744:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758290.113746:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758290.113749:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758290.113750:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758290.113752:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787758290.113754:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758290.113755:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758290.113757:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758290.113763:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:2.0:1787758290.113766:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:0.0:1787758290.113768:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758290.113772:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758290.113776:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758290.113778:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113781:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758290.113784:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758290.113785:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00000001:0.0:1787758290.113787:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113792:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.113793:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:0.0:1787758290.113798:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758290.113799:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758290.113803:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.113804:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758290.113809:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758290.113812:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758290.113814:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.113815:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113821:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.113824:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306017985 00000100:00100000:0.0:1787758290.113829:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f800 x1874588659733376/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758290.113838:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758290.113841:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.113842:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787758290.113844:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758290.113850:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758290.113853:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758290.113856:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306017985 00000100:00000001:0.0:1787758290.113857:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758290.113860:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758290.113865:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758290.113868:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11344 00000100:00000001:2.0:1787758290.113869:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:0.0:1787758290.113871:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758290.113874:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758290.113875:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758290.113877:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758290.113878:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758290.113880:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:0.0:1787758290.113882:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758290.113884:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758290.113885:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758290.113890:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758290.113894:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758290.113896:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758290.113898:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758290.113900:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758290.113907:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.113908:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113909:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758290.113916:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758290.113917:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00000001:0.0:1787758290.113919:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113923:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758290.113927:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758290.113931:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113933:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:2.0:1787758290.113934:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758290/real 1787758290] req@ffff8ffb4790d500 x1874588659733120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:0.0:1787758290.113935:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758290.113939:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113946:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758290.113955:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790d880 x1874588659733504/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758290.113964:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758290.113969:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758290.113975:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787758290.113981:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758290.113985:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758290.113990:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758290.113993:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758290.113996:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.113999:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758290.114001:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758290.114009:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d500 x1874588659733120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758290.114017:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114020:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758290.114023:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114024:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758290.114026:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114028:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758290.114030:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758290.114032:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758290.114036:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758290.114039:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758290.114042:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659733120:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758290.114047:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114050:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758290.114052:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758290.114054:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758290.114058:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758290.114060:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758290.114062:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758290.114066:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758290.114067:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758290.114070:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758290.114071:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114073:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114080:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758290.114081:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114083:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758290.114085:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758290.114087:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758290.114088:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758290.114090:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.114092:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758290.114094:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758290.114096:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758290.114099:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733248:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758290.114102:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758290.114104:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758290.114106:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.114107:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758290.114111:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114117:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758290.114119:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133081856 : -123165576469760 : ffff8ffb4790e300) 00000100:00000001:2.0:1787758290.114124:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758290.114138:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758290.114143:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758290.114145:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758290.114147:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114148:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758290.114151:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114155:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306017293 00000100:00000001:2.0:1787758290.114158:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306017985 00000100:00000001:2.0:1787758290.114161:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306017293 00000100:00000001:2.0:1787758290.114163:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306017985 00000100:00000001:2.0:1787758290.114165:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758290.114167:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114169:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758290.114173:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758290/real 1787758290] req@ffff8ffb4790e300 x1874588659733248/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758290.114180:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758290.114181:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758290.114216:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758290.114219:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758290.114221:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114222:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758290.114224:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758290.114228:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e300 x1874588659733248/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758290.114235:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114237:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758290.114240:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114242:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758290.114243:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114244:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758290.114256:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758290.114259:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758290.114261:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758290.114263:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758290.114266:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733248:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758290.114270:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758290.114271:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758290.114273:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.114275:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758290.114277:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758290.114279:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758290.114281:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733376:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758290.114284:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758290.114286:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758290.114287:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.114289:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758290.114292:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114297:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758290.114298:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133087232 : -123165576464384 : ffff8ffb4790f800) 00000100:00000001:2.0:1787758290.114302:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758290.114312:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758290.114317:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758290.114319:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758290.114320:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114322:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758290.114325:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114327:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306017291 00000100:00000001:2.0:1787758290.114330:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306017985 00000100:00000001:2.0:1787758290.114332:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306017291 00000100:00000001:2.0:1787758290.114334:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306017985 00000100:00000001:2.0:1787758290.114337:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758290.114339:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114341:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758290.114347:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758290/real 1787758290] req@ffff8ffb4790f800 x1874588659733376/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758290.114359:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758290.114361:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758290.114364:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758290.114366:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758290.114368:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114372:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758290.114374:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758290.114380:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f800 x1874588659733376/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758290.114391:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114395:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758290.114399:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114402:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758290.114404:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114407:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758290.114423:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758290.114427:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758290.114430:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758290.114432:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758290.114435:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733376:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758290.114439:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758290.114441:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758290.114442:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.114444:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758290.114446:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758290.114448:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758290.114451:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758290.114454:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758290.114456:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758290.114457:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758290.114459:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758290.114462:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114468:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758290.114469:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133079168 : -123165576472448 : ffff8ffb4790d880) 00000100:00000001:2.0:1787758290.114493:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758290.114509:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758290.114514:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758290.114517:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758290.114518:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114520:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758290.114522:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114526:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306017291 00000100:00000001:2.0:1787758290.114528:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306017985 00000100:00000001:2.0:1787758290.114530:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306017291 00000100:00000001:2.0:1787758290.114533:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306017985 00000100:00000001:2.0:1787758290.114535:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758290.114536:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114538:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758290.114542:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758290/real 1787758290] req@ffff8ffb4790d880 x1874588659733504/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758290.114549:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758290.114550:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758290.114552:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758290.114554:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758290.114555:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114557:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758290.114558:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758290.114561:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d880 x1874588659733504/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758290.114567:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114569:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758290.114571:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114573:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758290.114574:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114576:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758290.114586:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758290.114588:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758290.114591:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758290.114593:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758290.114596:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733504:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758290.114600:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114601:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758290.114602:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758290.114604:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758290.114606:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758290.114608:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758290.114609:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758290.114612:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758290.114613:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758290.114615:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758290.114616:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114618:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758290.114619:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758290.114620:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758290.114622:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758290.114623:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758290.114624:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758290.114625:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758290.114626:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758290.114628:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758290.114629:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114630:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758290.114631:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758290.114632:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758290.114633:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758290.114634:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758290.114635:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758290.114637:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758290.114638:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758290.114639:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758290.114640:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114642:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.114645:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758290.114646:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.114648:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758290.114650:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758290.689341:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758290.689350:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758290.689353:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758290.689359:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758290.689363:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758290.689368:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.689371:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758290.689374:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758290.689378:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758290.881374:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758290.881383:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758290.881392:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758290.881396:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758290.881500:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758290.881509:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758290.881516:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758291.713298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758291.713306:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758291.713308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758291.713313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758291.713318:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758291.713321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758291.713324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758291.713327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758291.713329:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758291.905292:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758291.905298:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758291.905304:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758291.905305:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758291.905332:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758291.905335:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758291.905338:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758292.737331:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758292.737340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758292.737343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758292.737349:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758292.737353:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758292.737359:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758292.737364:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758292.737368:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758292.737373:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758292.929320:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758292.929329:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758292.929338:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758292.929340:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758292.929377:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758292.929381:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758292.929385:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758293.761247:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758293.761253:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758293.761255:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758293.761259:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758293.761262:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758293.761265:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758293.761267:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758293.761269:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758293.761271:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758293.953273:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758293.953279:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758293.953284:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758293.953286:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758293.953348:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758293.953352:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758293.953354:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758294.785295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758294.785302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758294.785304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758294.785308:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758294.785311:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758294.785314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758294.785316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758294.785318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758294.785320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758294.977283:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758294.977299:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758294.977305:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758294.977309:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758294.977312:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758294.977315:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758294.977318:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758294.977320:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758294.977323:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758294.977326:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758294.977374:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758294.977377:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758294.977380:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758295.233349:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758295.233362:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758295.233365:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758295.233369:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758295.233373:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758295.233381:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11349 00000100:00080000:2.0:1787758295.233411:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758295.233414:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758295.233416:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758295.233420:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758295.233422:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758295.233424:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758295.233427:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758295.233429:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758295.233430:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758295.233434:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758295.233436:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758295.233439:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758295.233442:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758295.233445:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233447:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758295.233449:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758295.233460:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758295.233461:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233464:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758295.233466:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758295.233471:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233482:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758295.233491:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659733632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758295.233501:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758295.233505:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233522:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758295.233525:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758295.233528:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758295.233529:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758295.233531:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758295.233533:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758295.233535:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11349 00000100:00080000:2.0:1787758295.233538:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758295.233540:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758295.233541:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758295.233544:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758295.233545:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758295.233547:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758295.233549:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758295.233551:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758295.233552:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758295.233555:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758295.233557:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.233560:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787758295.233560:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758295.233565:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758295.233566:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787758295.233570:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.233571:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758295.233573:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758295.233575:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758295.233579:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:2.0:1787758295.233580:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758295.233584:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00000001:2.0:1787758295.233585:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.233590:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758295.233590:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.233597:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758295.233597:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758295.233601:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233603:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758295.233606:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787758295.233607:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659733632:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787758295.233608:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.233616:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787758295.233616:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.233624:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00100000:2.0:1787758295.233626:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659733760/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758295.233628:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.233635:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758295.233640:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758295.233642:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.233643:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233645:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758295.233649:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758295.233652:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758295.233654:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758295.233655:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758295.233658:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758295.233661:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787758295.233661:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11349 00000100:00080000:2.0:1787758295.233663:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758295.233665:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:2.0:1787758295.233667:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758295.233670:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758295.233673:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:2.0:1787758295.233673:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758295.233676:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758295.233678:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758295.233680:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758295.233682:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758295.233683:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758295.233686:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758295.233688:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758295.233690:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758295.233692:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758295.233694:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233696:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758295.233698:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758295.233701:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233702:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758295.233706:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758295.233707:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233708:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758295.233711:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000800:00000100:1.0:1787758295.233714:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:2.0:1787758295.233714:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233722:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758295.233727:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659733888/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758295.233729:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758295.233734:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758295.233738:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233739:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758295.233743:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.233744:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758295.233747:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758295.233751:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758295.233752:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758295.233756:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758295.233759:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306023105 00000100:00000001:2.0:1787758295.233760:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758295.233763:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758295.233766:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758295.233768:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11349 00000100:00080000:2.0:1787758295.233771:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758295.233772:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758295.233773:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758295.233776:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758295.233779:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758295.233780:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758295.233781:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306023105 00000020:00000001:2.0:1787758295.233782:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758295.233789:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758295.233790:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787758295.233792:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758295.233795:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758295.233798:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758295.233800:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758295.233802:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758295.233805:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758295.233807:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758295.233813:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758295.233814:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233820:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758295.233821:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:2.0:1787758295.233824:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758295.233829:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233832:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758295.233835:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758295/real 1787758295] req@ffff8ffb47a38000 x1874588659733632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787758295.233839:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758295.233842:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233846:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758295.233849:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758295.233853:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.233862:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787758295.233864:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758295.233887:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00100000:2.0:1787758295.233893:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659734016/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758295.233897:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758295.233904:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758295.233910:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758295.233912:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758295.233916:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758295.233918:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.233922:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758295.233926:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758295.233955:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659733632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758295.233963:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.233967:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758295.233971:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.233972:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758295.233974:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.233976:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758295.233978:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758295.233980:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758295.233985:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758295.233987:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758295.233990:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659733632:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758295.233995:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.233997:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758295.233998:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758295.234000:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758295.234003:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758295.234004:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758295.234006:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758295.234010:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758295.234012:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758295.234014:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758295.234016:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234018:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234024:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758295.234026:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234028:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758295.234030:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758295.234031:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758295.234032:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758295.234034:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.234036:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758295.234038:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758295.234040:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758295.234042:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733760:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758295.234045:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758295.234047:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758295.234048:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.234049:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758295.234052:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234058:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758295.234059:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:1.0:1787758295.234063:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758295.234075:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758295.234079:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758295.234081:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758295.234083:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234085:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758295.234087:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234103:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306022413 00000100:00000001:1.0:1787758295.234106:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306023105 00000100:00000001:1.0:1787758295.234112:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306022413 00000100:00000001:1.0:1787758295.234114:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306023105 00000100:00000001:1.0:1787758295.234116:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758295.234118:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234120:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758295.234124:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758295/real 1787758295] req@ffff8ffb47a38380 x1874588659733760/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758295.234130:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758295.234132:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758295.234134:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758295.234135:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758295.234136:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234138:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758295.234139:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758295.234142:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659733760/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758295.234148:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234150:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758295.234152:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234153:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758295.234154:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234155:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758295.234166:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758295.234168:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758295.234170:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758295.234173:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758295.234176:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733760:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758295.234180:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758295.234222:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758295.234223:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.234226:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758295.234228:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758295.234230:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758295.234233:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733888:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758295.234237:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758295.234239:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758295.234241:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.234243:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758295.234246:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234253:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758295.234254:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:1.0:1787758295.234258:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758295.234272:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758295.234278:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758295.234282:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758295.234284:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234286:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758295.234291:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234307:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306022411 00000100:00000001:1.0:1787758295.234311:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306023105 00000100:00000001:1.0:1787758295.234315:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306022411 00000100:00000001:1.0:1787758295.234319:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306023105 00000100:00000001:1.0:1787758295.234322:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758295.234325:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234329:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758295.234336:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758295/real 1787758295] req@ffff8ffb47a39880 x1874588659733888/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758295.234369:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758295.234371:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758295.234375:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758295.234377:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758295.234378:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234380:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758295.234382:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758295.234405:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659733888/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758295.234430:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234432:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758295.234434:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234435:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758295.234436:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234438:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758295.234447:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758295.234449:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758295.234451:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758295.234453:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758295.234456:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659733888:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758295.234460:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758295.234461:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758295.234462:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.234465:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758295.234466:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758295.234468:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758295.234470:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758295.234473:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758295.234474:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758295.234476:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758295.234478:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758295.234481:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234486:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758295.234487:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787758295.234491:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758295.234502:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758295.234506:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758295.234509:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758295.234510:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234511:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758295.234524:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234526:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306022411 00000100:00000001:1.0:1787758295.234529:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306023105 00000100:00000001:1.0:1787758295.234531:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306022411 00000100:00000001:1.0:1787758295.234533:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306023105 00000100:00000001:1.0:1787758295.234535:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758295.234536:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234538:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758295.234541:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758295/real 1787758295] req@ffff8ffb47a3a680 x1874588659734016/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758295.234547:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758295.234547:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758295.234549:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758295.234551:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758295.234552:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234553:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758295.234555:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758295.234558:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659734016/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758295.234563:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234565:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758295.234567:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234569:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758295.234570:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234571:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758295.234581:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758295.234583:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758295.234586:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758295.234588:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758295.234590:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734016:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758295.234593:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234594:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758295.234595:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758295.234597:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758295.234598:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758295.234599:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758295.234600:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758295.234602:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758295.234603:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758295.234605:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758295.234606:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234608:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758295.234609:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758295.234610:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758295.234612:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758295.234613:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758295.234614:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758295.234615:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758295.234616:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758295.234618:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758295.234619:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234621:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758295.234621:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758295.234622:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758295.234623:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758295.234624:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758295.234625:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758295.234627:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758295.234628:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758295.234630:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758295.234632:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234633:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758295.234636:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758295.234638:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758295.234639:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758295.234641:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758295.809311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758295.809320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758295.809323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758295.809329:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758295.809333:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758295.809338:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758295.809342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758295.809345:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758295.809348:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758296.001264:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758296.001270:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758296.001275:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758296.001276:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758296.001290:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758296.001292:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758296.001294:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758296.833297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758296.833304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758296.833306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758296.833310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758296.833314:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758296.833318:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758296.833321:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758296.833324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758296.833328:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758297.025343:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758297.025367:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758297.025375:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758297.025377:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758297.025403:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758297.025407:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758297.025411:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758297.857284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758297.857293:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758297.857295:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758297.857300:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758297.857304:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758297.857308:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758297.857311:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758297.857314:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758297.857317:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758298.049367:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758298.049383:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758298.049388:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758298.049395:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758298.049404:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758298.049409:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758298.049413:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758298.049418:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758298.049428:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758298.049430:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758298.049525:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758298.049532:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758298.049536:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758298.881316:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758298.881350:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758298.881354:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758298.881377:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758298.881383:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758298.881389:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758298.881393:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758298.881398:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758298.881403:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758299.073339:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758299.073347:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758299.073373:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758299.073375:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758299.073468:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758299.073476:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758299.073484:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758299.905298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758299.905321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758299.905324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758299.905331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758299.905335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758299.905339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758299.905342:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758299.905361:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758299.905364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758300.097389:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758300.097398:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758300.097407:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758300.097410:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758300.097449:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758300.097453:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758300.097459:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758300.353328:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758300.353342:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758300.353345:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758300.353350:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758300.353356:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758300.353364:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11355 00000100:00080000:2.0:1787758300.353378:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758300.353382:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758300.353385:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758300.353390:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758300.353394:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758300.353409:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758300.353412:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758300.353415:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758300.353417:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758300.353424:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758300.353427:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758300.353430:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758300.353435:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758300.353439:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353442:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758300.353445:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758300.353459:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758300.353461:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353465:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758300.353468:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758300.353473:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353488:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758300.353500:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659734144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758300.353523:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758300.353529:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353549:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758300.353553:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758300.353558:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758300.353560:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758300.353562:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758300.353565:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758300.353568:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11355 00000100:00080000:2.0:1787758300.353573:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758300.353576:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758300.353578:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758300.353581:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758300.353583:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758300.353586:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758300.353588:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758300.353591:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758300.353593:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758300.353596:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758300.353599:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758300.353601:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758300.353608:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758300.353611:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353614:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758300.353621:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758300.353625:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353628:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758300.353635:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758300.353638:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353640:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758300.353643:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758300.353646:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353652:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758300.353661:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659734272/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758300.353671:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758300.353675:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.353676:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758300.353680:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758300.353692:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758300.353695:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:2.0:1787758300.353698:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758300.353705:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758300.353705:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758300.353719:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758300.353726:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758300.353731:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:2.0:1787758300.353731:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11355 00000100:00000001:1.0:1787758300.353745:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758300.353748:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758300.353761:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758300.353765:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787758300.353768:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758300.353778:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758300.353791:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758300.353792:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758300.353804:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758300.353808:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758300.353817:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00100000:1.0:1787758300.353819:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659734144:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758300.353825:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758300.353831:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758300.353839:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:2.0:1787758300.353847:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787758300.353851:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787758300.353852:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758300.353861:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758300.353862:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758300.353873:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758300.353874:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758300.353887:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353890:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758300.353893:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758300.353897:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787758300.353898:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353918:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758300.353930:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758300.353932:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353934:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758300.353936:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758300.353940:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.353947:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758300.353949:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.353955:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:1.0:1787758300.353966:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:2.0:1787758300.353973:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659734400/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758300.353986:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758300.353988:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.353997:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758300.354002:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758300.354010:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758300.354014:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758300.354018:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758300.354025:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758300.354031:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11355 00000100:00080000:2.0:1787758300.354041:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758300.354043:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758300.354056:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758300.354063:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758300.354070:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758300.354072:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758300.354076:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758300.354081:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000020:00000001:2.0:1787758300.354082:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758300.354094:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758300.354095:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758300.354106:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.354106:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758300.354119:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758300.354129:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758300.354138:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758300.354143:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306028225 00000100:00000001:2.0:1787758300.354146:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758300.354153:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758300.354159:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758300.354163:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354166:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758300.354166:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758300.354176:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758300.354182:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.354229:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758300.354237:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758300.354239:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.354242:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758300.354245:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758300.354248:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.354255:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354258:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306028226 00000100:00100000:2.0:1787758300.354267:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659734528/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758300.354282:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:2.0:1787758300.354328:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758300.354331:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.354337:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354338:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758300.354354:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354359:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758300.354374:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758300/real 1787758300] req@ffff8ffb47a39500 x1874588659734144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758300.354393:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758300.354396:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758300.354404:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758300.354408:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758300.354411:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354414:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758300.354417:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758300.354424:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659734144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758300.354435:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354440:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758300.354444:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354446:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758300.354448:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354451:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758300.354454:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758300.354459:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758300.354464:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758300.354468:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758300.354472:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659734144:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758300.354479:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758300.354481:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758300.354484:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758300.354487:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758300.354491:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758300.354494:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758300.354498:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734272:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758300.354506:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758300.354509:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758300.354511:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758300.354514:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758300.354519:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354528:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758300.354530:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00000001:1.0:1787758300.354537:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758300.354557:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758300.354564:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758300.354567:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758300.354569:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354571:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758300.354575:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354581:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306027534 00000100:00000001:1.0:1787758300.354585:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306028225 00000100:00000001:1.0:1787758300.354588:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306027534 00000100:00000001:1.0:1787758300.354591:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306028226 00000100:00000001:1.0:1787758300.354594:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758300.354596:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354599:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758300.354604:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758300/real 1787758300] req@ffff8ffb47a38a80 x1874588659734272/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758300.354614:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758300.354616:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758300.354618:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758300.354621:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758300.354623:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354625:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758300.354627:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758300.354631:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659734272/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758300.354649:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354651:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758300.354655:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354657:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758300.354658:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354660:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758300.354682:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758300.354685:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758300.354689:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758300.354692:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758300.354696:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734272:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758300.354702:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354705:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758300.354708:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758300.354711:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758300.354716:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758300.354719:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758300.354723:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758300.354729:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758300.354732:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758300.354738:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758300.354740:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354742:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758300.354743:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758300.354745:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758300.354748:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758300.354750:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758300.354751:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758300.354755:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758300.354756:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758300.354760:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758300.354761:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354764:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354774:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758300.354777:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354781:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758300.354783:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758300.354785:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758300.354788:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758300.354790:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758300.354792:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758300.354794:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758300.354797:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758300.354801:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734400:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758300.354805:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758300.354807:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758300.354809:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758300.354812:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758300.354815:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354821:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758300.354823:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787758300.354829:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758300.354842:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758300.354848:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758300.354851:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758300.354852:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354854:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758300.354858:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354862:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306027531 00000100:00000001:1.0:1787758300.354865:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306028225 00000100:00000001:1.0:1787758300.354868:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306027531 00000100:00000001:1.0:1787758300.354871:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306028226 00000100:00000001:1.0:1787758300.354874:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758300.354876:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.354878:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758300.354883:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758300/real 1787758300] req@ffff8ffb47a3a680 x1874588659734400/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758300.354902:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758300.354903:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758300.354907:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758300.354909:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758300.354910:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354912:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758300.354914:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758300.354919:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659734400/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758300.354928:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354930:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758300.354933:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354935:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758300.354937:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.354939:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758300.354957:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758300.354960:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758300.354963:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758300.354966:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758300.354970:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734400:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758300.354974:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758300.354976:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758300.354978:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758300.354981:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758300.354983:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758300.354986:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758300.354989:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758300.354993:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758300.354995:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758300.354997:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758300.355000:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758300.355003:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.355007:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758300.355009:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:1.0:1787758300.355014:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758300.355025:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758300.355030:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758300.355032:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758300.355034:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.355036:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758300.355039:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.355043:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306027531 00000100:00000001:1.0:1787758300.355046:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306028225 00000100:00000001:1.0:1787758300.355049:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306027531 00000100:00000001:1.0:1787758300.355052:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306028226 00000100:00000001:1.0:1787758300.355055:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758300.355057:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.355059:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758300.355064:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758300/real 1787758300] req@ffff8ffb47a39880 x1874588659734528/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758300.355072:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758300.355074:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758300.355076:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758300.355078:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758300.355079:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.355081:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758300.355083:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758300.355104:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659734528/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758300.355119:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.355122:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758300.355125:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.355127:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758300.355129:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.355131:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758300.355151:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758300.355154:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758300.355158:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758300.355161:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758300.355165:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734528:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758300.355169:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.355172:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758300.355173:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758300.355175:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758300.355178:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758300.355180:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758300.355215:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758300.355219:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758300.355221:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758300.355226:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758300.355229:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.355233:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758300.355235:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758300.355238:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758300.355241:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758300.355243:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758300.355245:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758300.355249:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758300.355251:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758300.355254:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758300.355256:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.355260:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758300.355266:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758300.355269:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758300.355273:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758300.355276:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758300.929257:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758300.929264:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758300.929265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758300.929269:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758300.929272:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758300.929276:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758300.929278:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758300.929280:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758300.929304:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758301.121400:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758301.121409:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758301.121415:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758301.121416:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758301.121453:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758301.121457:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758301.121460:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758301.953314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758301.953323:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758301.953326:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758301.953332:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758301.953337:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758301.953342:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758301.953345:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758301.953349:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758301.953353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758302.145321:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758302.145327:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758302.145331:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758302.145332:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758302.145369:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758302.145372:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758302.145376:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758302.977365:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758302.977372:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758302.977391:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758302.977397:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758302.977401:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758302.977405:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758302.977408:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758302.977411:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758302.977414:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758303.169309:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758303.169317:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758303.169324:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758303.169326:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758303.169402:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758303.169409:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758303.169415:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758304.001298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758304.001305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758304.001308:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758304.001313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758304.001317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758304.001321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758304.001324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758304.001327:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758304.001330:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758304.193295:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758304.193303:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758304.193310:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758304.193313:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758304.193316:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758304.193319:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758304.193322:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758304.193324:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758304.193327:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758304.193329:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758304.193349:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758304.193351:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758304.193354:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758305.025256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758305.025263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758305.025265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758305.025270:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758305.025274:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758305.025277:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758305.025279:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758305.025281:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758305.025283:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758305.217627:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758305.217634:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758305.217640:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758305.217642:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758305.217672:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758305.217675:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758305.217678:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758305.473409:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758305.473422:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787758305.473426:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758305.473430:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758305.473436:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758305.473445:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11360 00000100:00080000:0.0:1787758305.473458:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758305.473479:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758305.473482:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758305.473487:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758305.473490:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758305.473493:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787758305.473496:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787758305.473498:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758305.473500:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758305.473505:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758305.473507:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758305.473510:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758305.473513:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758305.473517:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473519:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758305.473522:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758305.473535:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758305.473537:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473540:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758305.473544:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758305.473549:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473564:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758305.473574:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790c380 x1874588659734656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758305.473586:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758305.473591:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473612:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758305.473617:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758305.473631:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787758305.473632:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758305.473634:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758305.473637:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758305.473640:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11360 00000100:00080000:0.0:1787758305.473644:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758305.473647:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758305.473648:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758305.473651:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758305.473652:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758305.473655:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787758305.473657:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758305.473659:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758305.473660:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758305.473663:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758305.473666:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758305.473667:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758305.473672:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758305.473674:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473676:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758305.473694:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758305.473697:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473700:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758305.473705:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758305.473707:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473709:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758305.473710:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758305.473712:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473717:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.473721:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:0.0:1787758305.473727:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f100 x1874588659734784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758305.473729:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758305.473736:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758305.473740:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758305.473746:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758305.473754:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:0.0:1787758305.473757:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758305.473760:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758305.473766:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758305.473767:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473798:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758305.473801:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659734656:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1787758305.473810:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758305.473812:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758305.473820:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:0.0:1787758305.473823:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787758305.473825:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473831:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:1.0:1787758305.473833:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:0.0:1787758305.473840:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787758305.473845:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473851:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758305.473863:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11360 00000100:00000001:1.0:1787758305.473868:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:0.0:1787758305.473870:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758305.473874:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:0.0:1787758305.473877:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758305.473883:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758305.473885:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:0.0:1787758305.473890:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758305.473895:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758305.473898:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787758305.473901:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758305.473904:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758305.473906:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758305.473912:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758305.473914:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758305.473917:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758305.473922:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758305.473925:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473928:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758305.473931:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758305.473935:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473942:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758305.473943:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:0.0:1787758305.473951:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758305.473956:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473960:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758305.473962:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:0.0:1787758305.473963:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758305.473971:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.473972:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758305.473978:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.473980:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.473982:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758305.473994:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758305.473996:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ce00 x1874588659734912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758305.474003:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306033345 00000100:00000001:0.0:1787758305.474018:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758305.474022:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474023:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787758305.474027:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758305.474034:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758305.474036:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306033345 00000100:00080000:0.0:1787758305.474042:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758305.474048:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758305.474049:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758305.474055:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758305.474059:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758305.474062:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11360 00000100:00080000:0.0:1787758305.474067:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758305.474069:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758305.474072:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758305.474072:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:0.0:1787758305.474082:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758305.474084:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758305.474086:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758305.474092:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758305.474092:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:0.0:1787758305.474100:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758305.474103:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758305.474106:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787758305.474114:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758305/real 1787758305] req@ffff8ffb4790c380 x1874588659734656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758305.474115:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758305.474124:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758305.474132:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758305.474138:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758305.474146:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.474154:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758305.474161:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758305.474166:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787758305.474166:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:0.0:1787758305.474175:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787758305.474176:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:0.0:1787758305.474182:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758305.474186:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.474188:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758305.474191:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758305.474194:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.474286:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474287:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758305.474296:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00100000:0.0:1787758305.474300:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ed80 x1874588659735040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758305.474302:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474329:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758305.474335:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758305.474353:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659734656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758305.474355:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758305.474362:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758305.474370:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474382:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474389:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758305.474393:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474396:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758305.474398:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474401:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758305.474405:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758305.474410:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758305.474416:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758305.474419:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758305.474424:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659734656:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758305.474430:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474434:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758305.474436:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758305.474439:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758305.474443:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758305.474447:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758305.474449:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758305.474455:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758305.474457:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758305.474460:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758305.474462:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474465:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474490:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758305.474492:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474494:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758305.474497:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758305.474499:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758305.474500:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758305.474502:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758305.474506:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758305.474508:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758305.474511:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758305.474515:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734784:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758305.474519:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758305.474521:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758305.474523:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758305.474525:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758305.474529:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474536:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758305.474538:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:1.0:1787758305.474544:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758305.474560:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758305.474565:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758305.474568:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758305.474570:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474572:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758305.474575:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474580:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306032654 00000100:00000001:1.0:1787758305.474584:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306033345 00000100:00000001:1.0:1787758305.474587:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306032654 00000100:00000001:1.0:1787758305.474589:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306033345 00000100:00000001:1.0:1787758305.474592:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758305.474594:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474596:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758305.474601:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758305/real 1787758305] req@ffff8ffb4790f100 x1874588659734784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758305.474620:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758305.474622:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758305.474625:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758305.474627:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758305.474629:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474631:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758305.474633:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758305.474637:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659734784/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758305.474645:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474648:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758305.474650:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474652:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758305.474653:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474655:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758305.474669:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758305.474672:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758305.474675:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758305.474677:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758305.474681:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734784:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758305.474685:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758305.474686:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758305.474688:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758305.474690:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758305.474692:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758305.474694:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758305.474697:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734912:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758305.474700:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758305.474702:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758305.474703:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758305.474705:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758305.474708:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474713:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758305.474714:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:1.0:1787758305.474719:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758305.474729:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758305.474733:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758305.474736:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758305.474738:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474740:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758305.474742:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474746:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306032651 00000100:00000001:1.0:1787758305.474749:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306033345 00000100:00000001:1.0:1787758305.474751:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306032651 00000100:00000001:1.0:1787758305.474754:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306033345 00000100:00000001:1.0:1787758305.474757:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758305.474759:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474761:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758305.474781:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758305/real 1787758305] req@ffff8ffb4790ce00 x1874588659734912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758305.474790:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758305.474792:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758305.474795:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758305.474801:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758305.474803:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474805:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758305.474807:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758305.474812:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659734912/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758305.474820:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474822:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758305.474825:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474827:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758305.474828:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474830:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758305.474843:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758305.474846:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758305.474849:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758305.474853:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758305.474857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659734912:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758305.474861:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758305.474863:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758305.474865:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758305.474868:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758305.474869:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758305.474872:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758305.474875:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758305.474879:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758305.474881:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758305.474883:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758305.474885:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758305.474889:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474894:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758305.474896:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:1.0:1787758305.474900:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758305.474912:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758305.474917:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758305.474919:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758305.474921:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474923:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758305.474927:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474931:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306032651 00000100:00000001:1.0:1787758305.474934:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306033345 00000100:00000001:1.0:1787758305.474937:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306032651 00000100:00000001:1.0:1787758305.474940:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306033345 00000100:00000001:1.0:1787758305.474943:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758305.474945:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.474947:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758305.474952:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758305/real 1787758305] req@ffff8ffb4790ed80 x1874588659735040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758305.474971:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758305.474972:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758305.474975:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758305.474977:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758305.474978:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474980:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758305.474982:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758305.474986:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659735040/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758305.474994:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.474997:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758305.474999:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.475001:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758305.475003:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.475004:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758305.475018:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758305.475021:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758305.475024:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758305.475026:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758305.475030:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735040:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758305.475035:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.475038:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758305.475040:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758305.475042:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758305.475044:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758305.475046:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758305.475047:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758305.475051:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758305.475052:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758305.475055:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758305.475056:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.475059:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758305.475061:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758305.475063:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758305.475066:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758305.475067:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758305.475069:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758305.475071:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758305.475072:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758305.475074:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758305.475076:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.475078:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758305.475079:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758305.475080:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758305.475082:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758305.475083:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758305.475084:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758305.475087:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758305.475088:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758305.475090:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758305.475092:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.475094:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758305.475099:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758305.475101:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758305.475105:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758305.475108:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758306.049327:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758306.049335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758306.049338:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758306.049344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758306.049349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758306.049354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758306.049358:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758306.049361:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758306.049364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758306.241387:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758306.241396:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758306.241409:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758306.241413:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758306.241475:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758306.241488:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758306.241498:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758307.073314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758307.073323:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758307.073326:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758307.073334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758307.073338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758307.073344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758307.073347:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758307.073350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758307.073354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758307.265384:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758307.265395:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758307.265399:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758307.265404:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758307.265411:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758307.265415:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758307.265418:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758307.265422:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758307.265427:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758307.265429:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758307.265538:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758307.265546:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758307.265552:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758308.097326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758308.097335:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758308.097338:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758308.097344:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758308.097349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758308.097353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758308.097357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758308.097360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758308.097364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758308.289360:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758308.289368:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758308.289375:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758308.289378:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758308.289423:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758308.289427:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758308.289430:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758309.121300:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758309.121313:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758309.121317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758309.121326:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758309.121332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758309.121339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758309.121343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758309.121348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758309.121353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758309.313304:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758309.313313:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758309.313320:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758309.313322:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758309.313349:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758309.313353:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758309.313359:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758310.145277:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758310.145284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758310.145287:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758310.145293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758310.145297:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758310.145301:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758310.145303:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758310.145306:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758310.145309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758310.337281:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758310.337289:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758310.337296:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758310.337298:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758310.337421:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758310.337438:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758310.337442:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758310.593271:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758310.593280:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787758310.593283:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758310.593286:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758310.593290:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758310.593296:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11365 00000100:00080000:0.0:1787758310.593305:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758310.593308:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758310.593310:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758310.593313:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758310.593315:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758310.593318:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787758310.593320:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787758310.593322:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758310.593323:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758310.593327:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758310.593329:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758310.593331:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758310.593333:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758310.593336:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593337:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758310.593339:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758310.593348:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758310.593349:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593359:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758310.593362:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758310.593366:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593376:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758310.593384:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790fb80 x1874588659735168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758310.593392:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758310.593396:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593412:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758310.593415:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758310.593419:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787758310.593420:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758310.593421:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758310.593424:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758310.593426:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11365 00000100:00080000:0.0:1787758310.593430:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758310.593431:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758310.593433:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758310.593434:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758310.593436:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758310.593438:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787758310.593440:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758310.593442:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758310.593443:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758310.593446:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758310.593448:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758310.593449:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758310.593453:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758310.593454:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593456:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758310.593461:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758310.593462:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00000001:0.0:1787758310.593463:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593468:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593470:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:0.0:1787758310.593471:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758310.593474:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593474:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758310.593479:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758310.593480:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758310.593480:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:0.0:1787758310.593484:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593485:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593489:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.593492:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00100000:0.0:1787758310.593496:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790d180 x1874588659735296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758310.593498:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758310.593505:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758310.593515:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659735168:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758310.593519:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758310.593521:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:0.0:1787758310.593523:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.593527:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:0.0:1787758310.593531:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.593531:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.593537:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:0.0:1787758310.593538:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787758310.593542:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758310.593544:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758310.593548:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758310.593550:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758310.593554:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758310.593558:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11365 00000100:00000001:1.0:1787758310.593558:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758310.593563:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00080000:0.0:1787758310.593566:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758310.593570:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758310.593570:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758310.593572:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758310.593575:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758310.593576:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758310.593579:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787758310.593581:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758310.593582:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758310.593583:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758310.593586:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758310.593588:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758310.593589:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758310.593600:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758310.593601:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593603:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758310.593607:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758310.593609:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593610:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758310.593615:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:0.0:1787758310.593616:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758310.593619:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593621:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758310.593622:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758310.593624:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593629:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1787758310.593630:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758310.593637:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790e680 x1874588659735424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758310.593638:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758310.593643:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593648:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758310.593648:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758310.593652:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593654:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758310.593656:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758310.593657:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758310.593659:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758310.593662:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758310.593663:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758310.593664:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306038465 00000100:00000001:0.0:1787758310.593665:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758310.593669:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11365 00000100:00080000:0.0:1787758310.593673:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758310.593674:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758310.593675:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758310.593677:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758310.593678:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758310.593678:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000020:00000001:0.0:1787758310.593681:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:0.0:1787758310.593684:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:0.0:1787758310.593686:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758310.593687:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758310.593687:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306038465 00000100:00080000:0.0:1787758310.593692:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758310.593695:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758310.593696:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758310.593696:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:0.0:1787758310.593699:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758310.593702:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593703:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758310.593704:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758310.593706:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593707:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593709:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758310.593710:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758310.593712:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593715:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758310.593716:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758310.593716:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758310.593718:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593722:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:0.0:1787758310.593724:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758310.593728:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ca80 x1874588659735552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758310.593733:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758310/real 1787758310] req@ffff8ffb4790fb80 x1874588659735168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758310.593737:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758310.593740:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758310.593743:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787758310.593753:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758310.593757:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758310.593762:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758310.593765:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758310.593767:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593770:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758310.593772:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758310.593777:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659735168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758310.593786:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593789:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758310.593792:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593794:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758310.593796:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593798:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758310.593800:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758310.593804:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758310.593809:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758310.593812:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758310.593817:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659735168:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758310.593824:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593826:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758310.593829:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758310.593832:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758310.593836:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758310.593839:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758310.593842:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758310.593846:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758310.593848:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758310.593851:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758310.593853:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593856:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593863:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758310.593866:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593870:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758310.593872:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758310.593874:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758310.593876:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758310.593877:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.593880:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758310.593882:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758310.593885:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758310.593889:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735296:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758310.593893:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758310.593895:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758310.593896:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.593897:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758310.593899:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593904:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758310.593905:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:1.0:1787758310.593909:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758310.593920:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758310.593924:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758310.593926:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758310.593927:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593928:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758310.593930:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593933:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306037773 00000100:00000001:1.0:1787758310.593936:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306038465 00000100:00000001:1.0:1787758310.593937:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306037773 00000100:00000001:1.0:1787758310.593939:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306038465 00000100:00000001:1.0:1787758310.593941:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758310.593942:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.593943:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758310.593946:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758310/real 1787758310] req@ffff8ffb4790d180 x1874588659735296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758310.593952:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758310.593952:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758310.593954:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758310.593955:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758310.593956:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593957:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758310.593959:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758310.593961:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659735296/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758310.593966:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593967:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758310.593969:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593970:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758310.593970:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.593971:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758310.593981:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758310.593982:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758310.593985:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758310.593987:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758310.593989:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735296:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758310.593992:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758310.593993:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758310.593994:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.593995:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758310.593996:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758310.593997:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758310.593999:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735424:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758310.594001:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758310.594002:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758310.594003:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.594005:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758310.594007:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594010:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758310.594011:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:1.0:1787758310.594014:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758310.594020:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758310.594022:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758310.594024:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758310.594025:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594026:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758310.594028:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594030:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306037770 00000100:00000001:1.0:1787758310.594032:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306038465 00000100:00000001:1.0:1787758310.594034:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306037770 00000100:00000001:1.0:1787758310.594035:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306038465 00000100:00000001:1.0:1787758310.594037:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758310.594038:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594039:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758310.594042:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758310/real 1787758310] req@ffff8ffb4790e680 x1874588659735424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758310.594048:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758310.594049:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758310.594050:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758310.594051:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758310.594052:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594053:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758310.594054:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758310.594057:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659735424/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758310.594062:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594063:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758310.594064:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594065:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758310.594066:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594067:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758310.594075:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758310.594076:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758310.594078:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758310.594079:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758310.594081:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735424:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758310.594083:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758310.594084:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758310.594085:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.594087:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758310.594088:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758310.594089:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758310.594091:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758310.594093:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758310.594094:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758310.594095:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758310.594096:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758310.594098:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594101:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758310.594102:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00000001:1.0:1787758310.594105:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758310.594110:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758310.594131:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758310.594133:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758310.594134:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594135:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758310.594137:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594139:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306037770 00000100:00000001:1.0:1787758310.594142:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306038465 00000100:00000001:1.0:1787758310.594143:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306037770 00000100:00000001:1.0:1787758310.594145:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306038465 00000100:00000001:1.0:1787758310.594146:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758310.594148:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594149:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758310.594152:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758310/real 1787758310] req@ffff8ffb4790ca80 x1874588659735552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758310.594156:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758310.594157:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758310.594159:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758310.594160:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758310.594161:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594162:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758310.594163:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758310.594166:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659735552/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758310.594170:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594172:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758310.594174:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594175:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758310.594176:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594178:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758310.594209:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758310.594212:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758310.594215:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758310.594217:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758310.594220:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735552:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758310.594224:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594226:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758310.594227:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758310.594228:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758310.594230:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758310.594231:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758310.594232:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758310.594234:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758310.594235:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758310.594237:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758310.594238:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594239:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758310.594240:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758310.594241:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758310.594242:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758310.594243:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758310.594244:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758310.594245:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758310.594245:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758310.594246:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758310.594247:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594248:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758310.594249:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758310.594249:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758310.594250:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758310.594251:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758310.594252:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758310.594253:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758310.594254:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758310.594256:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758310.594257:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594258:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758310.594260:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758310.594262:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758310.594263:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758310.594265:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758311.169313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758311.169346:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758311.169350:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758311.169360:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758311.169367:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758311.169374:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758311.169377:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758311.169381:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758311.169385:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758311.361341:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758311.361350:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758311.361359:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758311.361362:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758311.361425:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758311.361431:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758311.361435:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758312.193280:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758312.193289:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758312.193291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758312.193299:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758312.193303:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758312.193307:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758312.193310:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758312.193314:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758312.193317:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758312.385375:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758312.385388:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758312.385399:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758312.385402:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758312.385435:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758312.385439:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758312.385443:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758313.217253:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758313.217260:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758313.217262:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758313.217266:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758313.217269:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758313.217272:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758313.217274:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758313.217276:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758313.217278:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758313.409384:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758313.409400:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758313.409412:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758313.409415:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758313.409457:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758313.409462:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758313.409468:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758314.241297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758314.241304:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758314.241305:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758314.241309:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758314.241312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758314.241315:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758314.241317:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758314.241319:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758314.241321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758314.433241:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758314.433247:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758314.433253:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758314.433257:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758314.433259:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758314.433263:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758314.433266:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758314.433268:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758314.433270:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758314.433274:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758314.433337:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758314.433342:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758314.433346:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758315.265240:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758315.265246:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758315.265248:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758315.265251:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758315.265254:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758315.265257:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758315.265259:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758315.265261:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758315.265262:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758315.457392:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758315.457399:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758315.457406:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758315.457407:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758315.457513:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758315.457526:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758315.457530:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758315.713317:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758315.713332:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787758315.713337:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758315.713342:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758315.713350:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758315.713359:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11370 00000100:00080000:0.0:1787758315.713370:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758315.713374:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758315.713376:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758315.713380:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758315.713383:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758315.713402:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787758315.713405:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787758315.713407:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758315.713409:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758315.713413:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758315.713416:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758315.713419:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758315.713422:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758315.713425:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713427:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758315.713430:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758315.713450:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758315.713452:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713456:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758315.713459:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758315.713463:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713483:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758315.713493:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f480 x1874588659735680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758315.713507:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758315.713513:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713534:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758315.713539:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758315.713544:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787758315.713547:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758315.713549:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758315.713554:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758315.713559:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11370 00000100:00080000:0.0:1787758315.713566:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758315.713569:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758315.713572:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758315.713576:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758315.713578:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:0.0:1787758315.713579:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758315.713586:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758315.713587:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:0.0:1787758315.713591:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758315.713593:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758315.713599:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758315.713600:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758315.713603:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758315.713605:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758315.713611:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787758315.713613:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:0.0:1787758315.713619:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758315.713620:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758315.713624:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758315.713626:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758315.713633:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:1.0:1787758315.713635:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659735680:192.168.201.118@tcp:250:kworker.0 02000000:00000001:0.0:1787758315.713640:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.713644:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:0.0:1787758315.713645:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787758315.713653:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:0.0:1787758315.713654:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758315.713659:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:0.0:1787758315.713660:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758315.713665:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:0.0:1787758315.713666:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758315.713675:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758315.713675:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758315.713679:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713683:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758315.713685:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758315.713687:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713695:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758315.713702:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790dc00 x1874588659735808/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758315.713710:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1787758315.713711:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758315.713717:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:0.0:1787758315.713718:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713724:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.713728:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:0.0:1787758315.713741:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758315.713745:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758315.713747:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758315.713749:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758315.713752:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758315.713755:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11370 00000100:00080000:0.0:1787758315.713767:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758315.713770:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758315.713771:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758315.713773:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758315.713774:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758315.713776:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787758315.713778:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758315.713781:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758315.713785:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:1.0:1787758315.713787:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1787758315.713792:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758315.713796:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758315.713799:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758315.713803:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758315.713806:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:0.0:1787758315.713807:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713812:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758315.713815:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758315.713815:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758315.713821:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:0.0:1787758315.713822:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.713828:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1787758315.713829:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758315.713837:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758315.713837:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758315.713841:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713845:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758315.713846:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306043585 02000000:00000001:0.0:1787758315.713847:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758315.713851:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.713862:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758315.713868:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306043585 00000100:00000001:1.0:1787758315.713874:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:0.0:1787758315.713876:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.713878:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758315.713883:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758315.713886:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790c000 x1874588659735936/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758315.713887:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758315.713899:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758315/real 1787758315] req@ffff8ffb4790f480 x1874588659735680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758315.713900:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758315.713906:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.713912:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758315.713919:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758315.713926:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758315.713929:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000400:00000001:1.0:1787758315.713929:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758315.713934:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:0.0:1787758315.713935:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758315.713940:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:0.0:1787758315.713943:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758315.713945:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758315.713949:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.713952:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758315.713954:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758315.713958:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659735680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758315.713964:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11370 00000100:00000001:1.0:1787758315.713977:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:0.0:1787758315.713979:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758315.713981:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:0.0:1787758315.713990:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758315.713990:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758315.713996:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758315.714000:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758315.714002:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758315.714005:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758315.714008:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758315.714013:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:0.0:1787758315.714015:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758315.714016:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758315.714021:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758315.714043:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758315.714049:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758315.714051:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:0.0:1787758315.714055:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:1.0:1787758315.714056:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659735680:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758315.714061:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758315.714068:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:0.0:1787758315.714071:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758315.714073:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758315.714078:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714079:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1787758315.714086:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758315.714086:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758315.714092:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:0.0:1787758315.714094:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787758315.714099:0:2367:0:(genops.c:1217:class_import_put()) Process entered 02000000:00000001:0.0:1787758315.714100:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758315.714102:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:0.0:1787758315.714104:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758315.714109:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758315.714110:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000008:00000001:0.0:1787758315.714112:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758315.714114:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:0.0:1787758315.714116:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714119:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758315.714123:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:0.0:1787758315.714124:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758315.714126:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:0.0:1787758315.714128:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.714133:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758315.714134:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758315.714136:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:0.0:1787758315.714137:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714142:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714148:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758315.714149:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714150:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758315.714155:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758315.714158:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758315.714162:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758315.714163:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f480 x1874588659736064/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758315.714166:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758315.714172:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758315.714177:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758315.714180:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.714207:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758315.714212:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758315.714264:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735808:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758315.714271:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758315.714275:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758315.714278:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758315.714282:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758315.714287:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714298:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758315.714300:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133080064 : -123165576471552 : ffff8ffb4790dc00) 00000100:00000001:1.0:1787758315.714306:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758315.714326:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758315.714332:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758315.714335:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758315.714337:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714338:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758315.714342:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714346:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306042894 00000100:00000001:1.0:1787758315.714349:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306043585 00000100:00000001:1.0:1787758315.714351:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306042894 00000100:00000001:1.0:1787758315.714354:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306043585 00000100:00000001:1.0:1787758315.714357:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758315.714359:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714361:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758315.714366:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758315/real 1787758315] req@ffff8ffb4790dc00 x1874588659735808/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758315.714376:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758315.714377:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758315.714380:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758315.714381:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758315.714383:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714385:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758315.714386:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758315.714390:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790dc00 x1874588659735808/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758315.714397:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714400:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758315.714402:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714403:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758315.714404:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714406:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758315.714418:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758315.714421:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758315.714425:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758315.714427:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758315.714430:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735808:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758315.714434:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758315.714435:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758315.714437:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758315.714439:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758315.714440:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758315.714442:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758315.714446:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758315.714449:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758315.714451:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758315.714453:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758315.714455:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758315.714458:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714463:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758315.714464:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133072896 : -123165576478720 : ffff8ffb4790c000) 00000100:00000001:1.0:1787758315.714468:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758315.714478:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758315.714482:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758315.714484:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758315.714485:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714487:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758315.714490:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714492:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306042891 00000100:00000001:1.0:1787758315.714495:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306043585 00000100:00000001:1.0:1787758315.714498:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306042891 00000100:00000001:1.0:1787758315.714500:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306043585 00000100:00000001:1.0:1787758315.714502:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758315.714503:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714505:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758315.714509:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758315/real 1787758315] req@ffff8ffb4790c000 x1874588659735936/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758315.714517:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758315.714519:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758315.714521:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758315.714522:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758315.714524:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714525:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758315.714527:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758315.714531:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c000 x1874588659735936/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758315.714544:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714546:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758315.714549:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714550:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758315.714551:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714553:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758315.714563:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758315.714565:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758315.714568:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758315.714570:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758315.714572:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659735936:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758315.714576:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714578:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758315.714580:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758315.714582:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758315.714584:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758315.714585:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758315.714586:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758315.714589:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758315.714591:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758315.714593:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758315.714594:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714596:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758315.714597:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758315.714598:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758315.714601:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758315.714602:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758315.714603:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758315.714605:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758315.714606:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758315.714607:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758315.714608:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714610:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714613:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758315.714615:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714617:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758315.714619:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758315.714620:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758315.714622:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758315.714623:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758315.714625:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758315.714627:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758315.714629:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758315.714631:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758315.714635:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758315.714636:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758315.714638:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758315.714640:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758315.714642:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714646:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758315.714648:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133086336 : -123165576465280 : ffff8ffb4790f480) 00000100:00000001:1.0:1787758315.714651:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758315.714660:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758315.714664:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758315.714666:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758315.714668:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714669:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758315.714672:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714675:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306042891 00000100:00000001:1.0:1787758315.714678:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306043585 00000100:00000001:1.0:1787758315.714680:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306042891 00000100:00000001:1.0:1787758315.714682:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306043585 00000100:00000001:1.0:1787758315.714685:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758315.714687:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714690:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758315.714696:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758315/real 1787758315] req@ffff8ffb4790f480 x1874588659736064/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758315.714719:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758315.714721:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758315.714725:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758315.714727:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758315.714730:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714733:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758315.714735:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758315.714741:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f480 x1874588659736064/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758315.714753:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714757:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758315.714761:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714763:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758315.714765:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714768:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758315.714782:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758315.714784:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758315.714788:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758315.714790:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758315.714794:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736064:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758315.714798:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714800:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758315.714801:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758315.714803:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758315.714806:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758315.714807:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758315.714808:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758315.714811:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758315.714812:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758315.714815:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758315.714816:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714818:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758315.714822:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758315.714823:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758315.714825:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758315.714827:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758316.289275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758316.289282:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758316.289284:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758316.289289:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758316.289293:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758316.289297:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758316.289299:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758316.289302:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758316.289304:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758316.481261:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758316.481266:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758316.481271:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758316.481272:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758316.481289:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758316.481290:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758316.481293:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758317.313310:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758317.313319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758317.313323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758317.313330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758317.313335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758317.313339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758317.313343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758317.313347:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758317.313350:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758317.505313:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758317.505325:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758317.505328:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758317.505333:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758317.505340:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758317.505343:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758317.505347:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758317.505351:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758317.505355:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758317.505357:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758317.505396:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758317.505399:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758317.505403:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758318.337282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758318.337337:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758318.337339:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758318.337345:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758318.337349:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758318.337353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758318.337356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758318.337370:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758318.337373:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758318.529312:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758318.529321:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758318.529328:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758318.529329:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758318.529455:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758318.529461:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758318.529466:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758319.361285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758319.361292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758319.361294:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758319.361298:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758319.361301:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758319.361305:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758319.361306:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758319.361309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758319.361311:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758319.553293:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758319.553301:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758319.553307:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758319.553308:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758319.553357:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758319.553363:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758319.553366:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758320.385259:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758320.385265:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758320.385267:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758320.385271:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758320.385274:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758320.385277:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758320.385278:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758320.385280:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758320.385282:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758320.577267:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758320.577273:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758320.577277:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758320.577278:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758320.577294:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758320.577296:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758320.577298:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758320.833333:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758320.833344:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758320.833348:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758320.833352:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758320.833357:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758320.833364:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11375 00000100:00080000:3.0:1787758320.833375:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758320.833379:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758320.833381:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758320.833385:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758320.833388:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758320.833392:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758320.833395:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758320.833397:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758320.833399:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758320.833404:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758320.833407:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758320.833410:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758320.833414:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758320.833417:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833419:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758320.833422:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758320.833434:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758320.833435:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833439:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758320.833441:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758320.833446:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833457:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758320.833468:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659736192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758320.833479:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758320.833483:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833502:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758320.833506:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758320.833512:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758320.833513:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758320.833515:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758320.833518:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758320.833521:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11375 00000100:00080000:3.0:1787758320.833524:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758320.833526:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758320.833527:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758320.833529:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758320.833531:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758320.833533:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758320.833535:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758320.833555:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758320.833556:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758320.833560:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758320.833574:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758320.833577:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758320.833582:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758320.833585:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833587:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758320.833592:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758320.833596:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833599:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758320.833605:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758320.833607:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833609:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758320.833611:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758320.833614:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833621:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.833629:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:3.0:1787758320.833629:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659736320/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758320.833638:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758320.833645:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758320.833650:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758320.833655:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833665:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758320.833666:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758320.833672:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.833674:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758320.833680:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.833681:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758320.833688:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758320.833696:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659736192:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787758320.833697:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758320.833704:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758320.833710:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787758320.833711:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758320.833719:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787758320.833720:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787758320.833726:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11375 02000000:00000001:1.0:1787758320.833727:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.833734:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787758320.833735:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758320.833740:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758320.833742:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833743:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758320.833751:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758320.833753:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758320.833755:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758320.833759:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758320.833761:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758320.833763:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758320.833766:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787758320.833768:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758320.833770:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:3.0:1787758320.833775:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.833779:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787758320.833780:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758320.833786:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758320.833789:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833806:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758320.833810:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758320.833813:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833816:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758320.833822:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758320.833824:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833830:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000800:00000100:1.0:1787758320.833831:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787758320.833833:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758320.833839:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833846:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.833849:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758320.833856:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758320.833860:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758320.833862:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659736448/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758320.833864:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758320.833874:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833880:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758320.833883:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306048705 00000100:00000001:3.0:1787758320.833902:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833908:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758320.833911:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758320.833914:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787758320.833915:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758320.833921:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758320.833924:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758320.833927:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306048705 00000100:00000001:3.0:1787758320.833928:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758320.833953:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11375 00000100:00000001:1.0:1787758320.833977:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758320.833986:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787758320.833988:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758320.833990:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758320.833999:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758320.834000:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758320.834008:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787758320.834016:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758320/real 1787758320] req@ffff8ffb45c30700 x1874588659736192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787758320.834018:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758320.834025:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758320.834031:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787758320.834043:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:3.0:1787758320.834043:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787758320.834048:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758320.834052:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758320.834056:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758320.834059:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758320.834062:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758320.834066:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834071:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787758320.834076:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758320.834077:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758320.834087:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758320.834090:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659736192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758320.834092:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758320.834100:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758320.834111:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834112:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758320.834117:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758320.834120:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000008:00100000:3.0:1787758320.834122:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758320.834128:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:3.0:1787758320.834130:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834134:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758320.834136:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834139:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834144:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:3.0:1787758320.834146:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758320.834147:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787758320.834153:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758320.834156:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787758320.834158:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758320.834163:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00080000:1.0:1787758320.834167:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 02000000:00000001:3.0:1787758320.834169:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834173:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787758320.834179:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758320.834181:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659736192:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758320.834189:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834192:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758320.834194:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00100000:3.0:1787758320.834243:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659736576/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758320.834248:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758320.834257:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787758320.834277:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000020:00000001:1.0:1787758320.834279:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:3.0:1787758320.834284:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758320.834285:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787758320.834290:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.834292:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758320.834297:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758320.834301:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758320.834303:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834307:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834315:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758320.834318:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834321:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758320.834324:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758320.834326:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758320.834329:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758320.834331:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.834334:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758320.834336:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758320.834338:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758320.834343:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736320:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758320.834347:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758320.834349:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758320.834351:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.834353:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758320.834356:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834363:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758320.834364:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787758320.834369:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758320.834383:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758320.834388:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758320.834390:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758320.834392:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834394:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758320.834397:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834401:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306048014 00000100:00000001:1.0:1787758320.834404:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306048705 00000100:00000001:1.0:1787758320.834406:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306048014 00000100:00000001:1.0:1787758320.834409:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306048705 00000100:00000001:1.0:1787758320.834411:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758320.834413:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834415:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758320.834419:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758320/real 1787758320] req@ffff8ffb45c32a00 x1874588659736320/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758320.834435:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758320.834437:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758320.834439:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758320.834441:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758320.834442:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834444:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758320.834445:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758320.834449:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659736320/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758320.834456:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834458:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758320.834461:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834462:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758320.834463:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834465:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758320.834477:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758320.834480:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758320.834483:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758320.834485:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758320.834488:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736320:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758320.834491:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758320.834493:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758320.834494:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.834496:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758320.834498:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758320.834500:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758320.834503:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736448:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758320.834506:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758320.834507:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758320.834510:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.834511:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758320.834514:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834519:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758320.834520:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787758320.834524:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758320.834552:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758320.834556:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758320.834559:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758320.834561:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834562:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758320.834565:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834569:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306048011 00000100:00000001:1.0:1787758320.834572:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306048705 00000100:00000001:1.0:1787758320.834575:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306048011 00000100:00000001:1.0:1787758320.834578:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306048705 00000100:00000001:1.0:1787758320.834580:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758320.834582:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834584:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758320.834589:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758320/real 1787758320] req@ffff8ffb45c31500 x1874588659736448/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758320.834597:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758320.834598:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758320.834600:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758320.834602:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758320.834603:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834605:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758320.834607:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758320.834611:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659736448/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758320.834619:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834621:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758320.834625:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834626:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758320.834628:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834630:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758320.834641:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758320.834644:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758320.834648:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758320.834650:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758320.834654:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736448:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758320.834657:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758320.834659:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758320.834660:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.834662:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758320.834664:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758320.834666:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758320.834669:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758320.834672:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758320.834674:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758320.834675:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758320.834677:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758320.834680:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834685:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758320.834686:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787758320.834690:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758320.834700:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758320.834704:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758320.834706:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758320.834708:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834710:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758320.834713:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834716:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306048011 00000100:00000001:1.0:1787758320.834719:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306048705 00000100:00000001:1.0:1787758320.834722:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306048011 00000100:00000001:1.0:1787758320.834725:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306048705 00000100:00000001:1.0:1787758320.834727:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758320.834730:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834732:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758320.834736:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758320/real 1787758320] req@ffff8ffb45c31180 x1874588659736576/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758320.834744:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758320.834745:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758320.834747:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758320.834749:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758320.834750:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834752:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758320.834753:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758320.834757:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659736576/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758320.834765:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834768:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758320.834770:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834773:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758320.834774:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834776:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758320.834787:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758320.834790:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758320.834792:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758320.834795:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758320.834799:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736576:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758320.834803:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834805:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758320.834807:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758320.834809:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758320.834811:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758320.834813:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758320.834815:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758320.834817:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758320.834819:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758320.834821:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758320.834823:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834825:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758320.834826:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758320.834828:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758320.834830:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758320.834832:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758320.834833:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758320.834835:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758320.834836:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758320.834838:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758320.834839:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834841:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758320.834842:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758320.834843:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758320.834845:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758320.834846:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758320.834847:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758320.834848:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758320.834850:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758320.834851:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758320.834852:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834854:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758320.834858:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758320.834859:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758320.834862:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758320.834863:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758321.409292:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758321.409300:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758321.409302:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758321.409307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758321.409310:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758321.409314:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758321.409316:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758321.409318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758321.409320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758321.601360:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758321.601372:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758321.601385:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758321.601389:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758321.601426:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758321.601431:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758321.601436:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758322.433256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758322.433263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758322.433265:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758322.433269:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758322.433272:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758322.433275:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758322.433277:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758322.433279:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758322.433281:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758322.625272:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758322.625281:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758322.625288:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758322.625290:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758322.625357:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758322.625364:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758322.625369:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758323.457285:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758323.457323:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758323.457326:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758323.457333:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758323.457338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758323.457355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758323.457359:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758323.457362:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758323.457367:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758323.649331:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758323.649342:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758323.649353:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758323.649356:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758323.649514:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758323.649520:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758323.649525:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758324.481283:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758324.481292:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758324.481309:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758324.481317:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758324.481324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758324.481331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758324.481335:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758324.481341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758324.481346:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758324.673308:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758324.673317:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758324.673326:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758324.673333:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758324.673337:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758324.673342:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758324.673347:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758324.673350:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758324.673356:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758324.673361:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758324.673393:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758324.673398:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758324.673403:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758325.505312:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758325.505322:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758325.505324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758325.505331:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758325.505336:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758325.505340:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758325.505344:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758325.505348:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758325.505351:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758325.697330:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758325.697341:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758325.697351:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758325.697354:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758325.697391:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758325.697395:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758325.697399:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758325.953293:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758325.953306:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758325.953309:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758325.953313:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758325.953319:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758325.953328:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11380 00000100:00080000:3.0:1787758325.953341:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758325.953345:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758325.953347:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758325.953353:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758325.953356:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758325.953360:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758325.953363:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758325.953366:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758325.953368:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758325.953374:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758325.953378:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758325.953381:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758325.953387:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758325.953391:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953394:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758325.953397:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758325.953409:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758325.953411:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953415:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758325.953418:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758325.953423:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953438:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758325.953450:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659736704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758325.953464:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758325.953469:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953497:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758325.953501:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758325.953506:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758325.953508:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758325.953523:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758325.953527:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758325.953530:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11380 00000100:00080000:3.0:1787758325.953535:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758325.953538:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758325.953540:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758325.953543:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758325.953545:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758325.953548:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758325.953550:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758325.953553:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758325.953555:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758325.953559:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758325.953562:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758325.953564:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758325.953571:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758325.953573:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953576:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758325.953581:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758325.953586:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953589:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758325.953595:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758325.953597:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953600:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758325.953603:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758325.953605:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953611:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758325.953618:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659736832/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758325.953620:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758325.953629:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758325.953637:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758325.953637:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758325.953643:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758325.953644:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.953649:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953653:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758325.953658:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787758325.953662:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787758325.953665:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787758325.953669:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787758325.953673:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953675:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758325.953681:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758325.953688:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:1.0:1787758325.953690:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659736704:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787758325.953698:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11380 00000100:00000001:1.0:1787758325.953707:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787758325.953709:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:1.0:1787758325.953715:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758325.953715:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758325.953723:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953723:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758325.953733:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:3.0:1787758325.953734:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758325.953740:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758325.953742:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758325.953745:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758325.953746:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953748:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758325.953754:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758325.953762:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758325.953765:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758325.953768:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758325.953771:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758325.953774:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953776:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758325.953779:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758325.953782:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953784:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758325.953790:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758325.953793:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787758325.953794:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.953800:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:3.0:1787758325.953802:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758325.953807:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758325.953812:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787758325.953813:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953824:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758325.953831:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659736960/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758325.953841:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758325.953875:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953878:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758325.953882:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758325.953887:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758325.953890:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758325.953892:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758325.953897:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758325.953900:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11380 00000100:00080000:3.0:1787758325.953908:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758325.953911:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758325.953913:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758325.953916:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758325.953919:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758325.953922:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758325.953924:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758325.953926:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758325.953929:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758325.953933:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758325.953934:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758325.953944:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306053825 00000100:00000001:3.0:1787758325.953945:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758325.953955:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758325.953962:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758325.953966:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787758325.953966:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953976:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758325.953980:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758325.953981:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306053825 00000008:00000001:3.0:1787758325.953987:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.953993:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.953994:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758325.954003:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758325.954004:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758325.954007:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758325.954009:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954015:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.954016:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758325.954023:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758325.954025:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758325.954029:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954038:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758325.954043:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954044:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787758325.954056:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758325.954064:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:3.0:1787758325.954064:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659737088/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758325.954071:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758325.954076:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758325.954082:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758325.954090:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758325.954097:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758325.954102:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758325.954104:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758325.954115:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758325.954117:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736832:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758325.954129:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758325.954132:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758325.954133:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758325.954139:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758325.954144:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758325.954150:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954160:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758325.954163:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:1.0:1787758325.954169:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758325.954249:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954255:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306053134 00000100:00000001:1.0:1787758325.954260:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306053825 00000100:00000001:1.0:1787758325.954263:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306053134 00000100:00000001:1.0:1787758325.954266:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306053825 00000100:00000001:1.0:1787758325.954269:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758325.954271:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954274:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758325.954275:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758325.954277:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758325.954280:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758325.954282:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758325.954284:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758325.954289:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736960:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758325.954293:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758325.954295:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758325.954297:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758325.954300:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758325.954304:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954310:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758325.954312:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787758325.954316:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758325.954331:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954336:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306053131 00000100:00000001:1.0:1787758325.954339:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306053825 00000100:00000001:1.0:1787758325.954342:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306053131 00000100:00000001:1.0:1787758325.954344:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306053825 00000100:00000001:1.0:1787758325.954347:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758325.954350:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954353:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954355:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758325.954359:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758325.954361:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787758325.954363:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758325.954365:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758325.954367:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758325.954369:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758325.954371:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758325.954373:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758325.954375:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758325.954377:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758325.954381:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758325.954385:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758325.954387:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758325.954389:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758325.954391:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758325.954394:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954398:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758325.954400:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787758325.954403:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758325.954413:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954416:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306053131 00000100:00000001:1.0:1787758325.954419:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306053825 00000100:00000001:1.0:1787758325.954422:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306053131 00000100:00000001:1.0:1787758325.954425:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306053825 00000100:00000001:1.0:1787758325.954427:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758325.954430:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954432:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954434:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758325.954437:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758325.954439:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787758325.954441:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758325.954443:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758325.954445:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758325.954448:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758326.529284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758326.529296:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758326.529301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758326.529310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758326.529317:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758326.529323:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758326.529328:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758326.529333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758326.529338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758326.721330:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758326.721340:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758326.721350:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758326.721354:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758326.721399:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758326.721423:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758326.721428:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758327.553336:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758327.553344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758327.553347:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758327.553352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758327.553356:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758327.553360:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758327.553363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758327.553365:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758327.553368:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758327.745281:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758327.745289:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758327.745291:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758327.745294:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758327.745298:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758327.745300:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758327.745301:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758327.745303:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758327.745306:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758327.745307:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758327.745360:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758327.745364:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758327.745366:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758328.577275:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758328.577280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758328.577282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758328.577286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758328.577289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758328.577292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758328.577294:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758328.577297:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758328.577299:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758328.769268:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758328.769273:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758328.769278:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758328.769278:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758328.769295:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758328.769297:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758328.769299:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787758329.025417:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1787758329.025456:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787758329.025470:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787758329.025481:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758329.025496:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758329.025498:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758329.025499:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787758329.025506:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787758329.025510:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758329.025512:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758329.025513:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758329.025513:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787758329.025516:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787758329.025520:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758329.025522:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758329.025524:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758329.025525:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:2.0:1787758329.025529:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:2.0:1787758329.025533:0:2356:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758329.025535:0:2356:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758329.025536:0:2356:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758329.025538:0:2356:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758329.025540:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758329.025554:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758329.025556:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758329.025570:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758325/real 1787758329] req@ffff8ffb45c31180 x1874588659737088/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758329.025587:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758329.025589:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758329.025596:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758329.025599:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758329.025600:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025604:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758329.025606:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758329.025611:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659737088/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758329.025619:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025622:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758329.025625:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025627:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758329.025628:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025630:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758329.025646:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758329.025650:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758329.025655:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758329.025658:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758329.025662:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737088:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758329.025667:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758329.025671:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758325/real 1787758329] req@ffff8ffb45c33480 x1874588659736832/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758329.025687:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758329.025688:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758329.025691:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758329.025693:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758329.025694:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025696:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758329.025698:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758329.025702:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659736832/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758329.025708:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025710:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758329.025712:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025714:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758329.025715:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025717:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758329.025730:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758329.025733:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758329.025736:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758329.025739:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758329.025743:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736832:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758329.025747:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758329.025751:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758325/real 1787758329] req@ffff8ffb45c33800 x1874588659736960/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758329.025760:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758329.025762:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758329.025765:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758329.025767:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758329.025768:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025770:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758329.025772:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758329.025776:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659736960/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758329.025788:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025807:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758329.025810:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025811:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758329.025813:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025814:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758329.025829:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758329.025832:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758329.025836:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758329.025839:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758329.025843:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659736960:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758329.025847:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758329.025851:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758325/real 1787758329] req@ffff8ffb45c30000 x1874588659736704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758329.025860:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758329.025861:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758329.025866:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758329.025867:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758329.025868:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025870:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758329.025871:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758329.025876:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659736704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758329.025882:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025884:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758329.025886:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025888:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758329.025890:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025892:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758329.025894:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758329.025896:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758329.025899:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758329.025902:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758329.025905:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659736704:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758329.025910:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025913:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758329.025915:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758329.025918:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758329.025925:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758329.025935:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758329.025938:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758329.025943:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758329.025945:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758329.025949:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758329.025951:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025952:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758329.025954:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758329.025955:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758329.025957:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758329.025958:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758329.025959:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758329.025961:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758329.025963:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758329.025965:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758329.025966:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025967:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758329.025969:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758329.025970:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758329.025973:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758329.025974:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758329.025975:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758329.025977:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758329.025978:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758329.025979:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758329.025981:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025983:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758329.025984:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758329.025985:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758329.025987:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758329.025988:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758329.025989:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758329.025991:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758329.025993:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758329.025994:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758329.025995:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.025999:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758329.026008:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758329.026010:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758329.026013:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758329.026014:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758329.601263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758329.601269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758329.601271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758329.601275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758329.601278:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758329.601281:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758329.601284:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758329.601286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758329.601288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758329.793288:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758329.793296:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758329.793318:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758329.793321:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758329.793349:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758329.793352:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758329.793355:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758330.625238:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758330.625245:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758330.625247:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758330.625252:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758330.625256:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758330.625259:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758330.625261:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758330.625262:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758330.625264:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758330.817261:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758330.817267:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758330.817272:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758330.817273:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758330.817314:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758330.817317:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758330.817320:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758331.073250:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758331.073261:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758331.073264:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758331.073268:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758331.073273:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758331.073281:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11385 00000100:00080000:3.0:1787758331.073292:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758331.073297:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758331.073299:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758331.073302:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758331.073305:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758331.073307:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758331.073309:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758331.073310:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758331.073311:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758331.073315:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758331.073317:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758331.073319:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758331.073322:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758331.073325:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073327:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758331.073329:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758331.073337:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758331.073338:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073340:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758331.073342:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758331.073346:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073356:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758331.073364:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659737216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758331.073372:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758331.073375:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073389:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758331.073393:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758331.073401:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758331.073403:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758331.073405:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758331.073408:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758331.073410:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11385 00000100:00080000:3.0:1787758331.073415:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758331.073417:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758331.073418:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758331.073421:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758331.073422:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758331.073424:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758331.073427:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758331.073429:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758331.073430:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758331.073430:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758331.073433:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758331.073434:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758331.073436:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073437:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758331.073438:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758331.073439:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758331.073441:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758331.073442:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758331.073444:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.073445:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758331.073446:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787758331.073448:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758331.073450:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00100000:3.0:1787758331.073450:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758331.073453:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758331.073455:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659737216:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758331.073455:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073460:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758331.073460:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758331.073462:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.073463:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758331.073463:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758331.073465:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758331.073466:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758331.073467:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.073468:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758331.073471:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.073472:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758331.073476:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659737344/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758331.073484:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758331.073486:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758331.073486:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073488:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787758331.073489:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073491:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787758331.073492:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758331.073495:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758331.073497:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758331.073498:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758331.073501:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758331.073503:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11385 00000100:00080000:3.0:1787758331.073506:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758331.073507:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758331.073509:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758331.073510:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758331.073512:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758331.073514:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758331.073516:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758331.073518:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758331.073519:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758331.073522:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758331.073524:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758331.073526:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787758331.073526:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758331.073529:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758331.073530:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073533:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758331.073535:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758331.073537:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000008:00000001:3.0:1787758331.073537:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073539:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073540:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758331.073542:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758331.073542:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758331.073544:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:3.0:1787758331.073544:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073546:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758331.073547:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758331.073548:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073550:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073552:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758331.073557:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659737472/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758331.073560:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306058945 00000100:00000001:3.0:1787758331.073564:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758331.073566:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073568:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073569:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787758331.073571:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758331.073574:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306058945 00000100:00080000:3.0:1787758331.073575:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758331.073576:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758331.073578:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787758331.073578:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758331.073581:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758331.073581:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758331.073583:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11385 00000100:00000001:1.0:1787758331.073584:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073586:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:3.0:1787758331.073587:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758331.073589:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758331.073590:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787758331.073592:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758331/real 1787758331] req@ffff8ffb45c31500 x1874588659737216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:3.0:1787758331.073592:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758331.073594:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758331.073596:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758331.073598:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758331.073601:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758331.073602:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:1.0:1787758331.073603:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758331.073604:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:3.0:1787758331.073605:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758331.073607:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073608:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787758331.073609:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758331.073610:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787758331.073612:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758331.073614:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073616:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758331.073616:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758331.073618:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758331.073619:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000008:00000001:3.0:1787758331.073620:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073621:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758331.073622:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758331.073626:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659737216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758331.073626:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758331.073627:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758331.073629:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758331.073631:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758331.073632:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073635:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758331.073636:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073638:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758331.073641:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758331.073641:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659737600/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758331.073643:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758331.073645:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073647:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758331.073648:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758331.073649:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758331.073650:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758331.073652:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787758331.073652:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758331.073656:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758331.073658:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758331.073667:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659737216:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758331.073672:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073674:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758331.073676:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758331.073679:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758331.073681:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758331.073684:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758331.073686:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758331.073690:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758331.073692:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758331.073694:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758331.073696:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073698:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073705:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758331.073707:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073709:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758331.073711:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758331.073712:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758331.073714:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758331.073715:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.073717:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758331.073719:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758331.073721:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758331.073725:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737344:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758331.073728:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758331.073730:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758331.073731:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.073733:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758331.073736:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073742:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758331.073744:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787758331.073748:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758331.073777:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758331.073782:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758331.073784:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758331.073786:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073787:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758331.073790:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073795:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306058253 00000100:00000001:1.0:1787758331.073798:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306058945 00000100:00000001:1.0:1787758331.073801:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306058253 00000100:00000001:1.0:1787758331.073803:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306058945 00000100:00000001:1.0:1787758331.073805:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758331.073807:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073809:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758331.073814:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758331/real 1787758331] req@ffff8ffb45c32a00 x1874588659737344/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758331.073824:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758331.073825:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758331.073828:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758331.073830:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758331.073832:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073833:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758331.073835:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758331.073839:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659737344/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758331.073847:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073849:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758331.073851:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073853:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758331.073854:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073856:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758331.073871:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758331.073874:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758331.073877:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758331.073879:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758331.073883:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737344:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758331.073887:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758331.073889:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758331.073890:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.073892:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758331.073894:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758331.073896:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758331.073899:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737472:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758331.073902:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758331.073904:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758331.073906:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.073908:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758331.073911:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073916:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758331.073918:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787758331.073921:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758331.073933:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758331.073938:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758331.073940:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758331.073941:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073943:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758331.073946:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073949:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306058250 00000100:00000001:1.0:1787758331.073952:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306058945 00000100:00000001:1.0:1787758331.073954:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306058250 00000100:00000001:1.0:1787758331.073956:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306058945 00000100:00000001:1.0:1787758331.073959:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758331.073960:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.073962:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758331.073966:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758331/real 1787758331] req@ffff8ffb45c30700 x1874588659737472/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758331.073974:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758331.073975:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758331.073978:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758331.073980:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758331.073981:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073984:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758331.073985:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758331.073989:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659737472/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758331.073995:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.073998:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758331.074000:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074002:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758331.074003:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074005:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758331.074018:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758331.074021:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758331.074024:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758331.074026:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758331.074030:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737472:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758331.074034:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758331.074035:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758331.074037:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.074039:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758331.074041:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758331.074043:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758331.074046:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758331.074049:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758331.074051:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758331.074054:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758331.074055:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758331.074058:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.074063:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758331.074065:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787758331.074068:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758331.074079:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758331.074084:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758331.074086:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758331.074087:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.074089:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758331.074091:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.074095:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306058250 00000100:00000001:1.0:1787758331.074098:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306058945 00000100:00000001:1.0:1787758331.074100:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306058250 00000100:00000001:1.0:1787758331.074102:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306058945 00000100:00000001:1.0:1787758331.074105:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758331.074107:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.074109:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758331.074115:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758331/real 1787758331] req@ffff8ffb45c30a80 x1874588659737600/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758331.074122:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758331.074123:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758331.074126:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758331.074127:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758331.074128:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074130:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758331.074131:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758331.074134:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659737600/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758331.074140:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074142:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758331.074144:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074146:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758331.074147:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074148:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758331.074161:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758331.074163:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758331.074167:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758331.074168:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758331.074171:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737600:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758331.074175:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074178:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758331.074179:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758331.074180:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758331.074216:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758331.074218:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758331.074219:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758331.074222:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758331.074223:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758331.074226:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758331.074227:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074228:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758331.074230:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758331.074231:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758331.074233:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758331.074234:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758331.074235:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758331.074236:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758331.074237:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758331.074239:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758331.074240:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074241:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758331.074243:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758331.074244:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758331.074245:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758331.074246:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758331.074247:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758331.074248:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758331.074249:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758331.074250:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758331.074252:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074253:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758331.074257:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758331.074258:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758331.074261:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758331.074263:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758331.649250:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758331.649256:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758331.649258:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758331.649261:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758331.649275:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758331.649278:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758331.649280:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758331.649282:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758331.649284:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758331.841352:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758331.841361:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758331.841369:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758331.841371:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758331.841448:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758331.841454:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758331.841459:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758332.673314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758332.673321:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758332.673323:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758332.673328:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758332.673332:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758332.673335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758332.673348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758332.673351:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758332.673354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758332.865308:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758332.865318:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758332.865326:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758332.865329:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758332.865396:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758332.865400:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758332.865405:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:2.0:1787758333.697314:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1787758333.697324:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758333.697327:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758333.697334:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:2.0:1787758333.697341:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758333.697345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758333.697348:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758333.697350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758333.697353:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758333.889356:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758333.889366:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758333.889374:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758333.889376:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758333.889450:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758333.889459:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758333.889466:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758334.721274:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758334.721281:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758334.721282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758334.721286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758334.721289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758334.721292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758334.721293:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758334.721295:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758334.721297:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758334.913275:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758334.913281:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758334.913285:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758334.913287:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758334.913289:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758334.913292:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758334.913302:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758334.913303:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758334.913305:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758334.913306:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758334.913357:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758334.913360:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758334.913362:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758335.745280:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758335.745288:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758335.745291:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758335.745296:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758335.745300:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758335.745304:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758335.745307:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758335.745309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758335.745312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758335.937284:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758335.937294:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758335.937302:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758335.937304:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758335.937375:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758335.937379:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758335.937384:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758336.193376:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758336.193389:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758336.193392:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758336.193397:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758336.193403:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758336.193411:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11390 00000100:00080000:2.0:1787758336.193425:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758336.193429:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758336.193432:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758336.193437:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758336.193440:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758336.193444:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758336.193447:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758336.193450:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758336.193453:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758336.193458:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758336.193462:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758336.193466:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758336.193470:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758336.193474:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193476:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758336.193479:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758336.193494:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758336.193497:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193501:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758336.193504:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758336.193509:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193523:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758336.193536:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659737728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758336.193549:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758336.193555:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193578:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758336.193582:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758336.193587:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758336.193589:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758336.193592:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758336.193595:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758336.193598:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11390 00000100:00080000:2.0:1787758336.193602:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758336.193605:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758336.193608:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758336.193612:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758336.193614:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758336.193617:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758336.193621:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758336.193625:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758336.193628:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758336.193630:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787758336.193634:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758336.193639:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758336.193641:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.193647:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787758336.193648:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758336.193654:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758336.193660:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758336.193661:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758336.193694:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193696:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758336.193703:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758336.193708:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193710:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758336.193716:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758336.193717:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758336.193719:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758336.193725:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758336.193726:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758336.193731:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758336.193734:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758336.193737:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193747:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758336.193748:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659737728:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1787758336.193759:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659737856/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758336.193761:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758336.193770:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758336.193776:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193778:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758336.193782:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758336.193784:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193792:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758336.193793:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758336.193796:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758336.193803:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758336.193806:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758336.193809:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758336.193812:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758336.193815:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787758336.193817:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11390 00000100:00000001:1.0:1787758336.193823:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00080000:2.0:1787758336.193826:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758336.193831:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758336.193833:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758336.193838:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758336.193843:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758336.193845:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758336.193847:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758336.193850:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758336.193853:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758336.193856:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758336.193860:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758336.193862:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758336.193865:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758336.193870:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758336.193873:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193875:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758336.193878:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758336.193881:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193887:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758336.193894:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758336.193896:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758336.193899:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758336.193900:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758336.193917:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758336.193920:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787758336.193922:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.193932:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758336.193934:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.193939:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.193947:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:2.0:1787758336.193948:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659737984/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758336.193955:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193967:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758336.193968:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306064065 00000100:00000001:2.0:1787758336.193974:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.193981:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758336.193985:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758336.193986:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:2.0:1787758336.193992:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758336.193997:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758336.193999:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306064065 00000100:00080000:2.0:1787758336.194001:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758336.194011:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758336.194013:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:2.0:1787758336.194018:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11390 00000100:00000001:1.0:1787758336.194023:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787758336.194026:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758336.194029:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.194032:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758336.194035:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758336.194038:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758336.194044:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758336.194048:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00100000:1.0:1787758336.194050:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758336/real 1787758336] req@ffff8ffb47a38380 x1874588659737728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:2.0:1787758336.194052:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758336.194062:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758336.194073:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758336.194079:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758336.194094:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758336.194106:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758336.194116:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000400:00000001:1.0:1787758336.194118:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:2.0:1787758336.194121:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:1.0:1787758336.194124:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:2.0:1787758336.194127:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194132:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758336.194133:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758336.194141:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000008:00100000:2.0:1787758336.194142:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758336.194148:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:2.0:1787758336.194150:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194156:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758336.194157:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194164:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:2.0:1787758336.194169:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758336.194175:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758336.194178:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659737728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758336.194180:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758336.194225:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758336.194228:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758336.194235:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194253:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758336.194254:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659738112/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758336.194263:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758336.194272:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194281:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758336.194284:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758336.194286:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758336.194292:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194294:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758336.194301:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194302:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758336.194314:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.194325:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758336.194344:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758336.194353:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659737728:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758336.194363:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194368:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758336.194371:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758336.194375:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758336.194381:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758336.194385:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758336.194388:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758336.194394:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758336.194397:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758336.194403:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758336.194406:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194411:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194422:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758336.194425:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194428:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758336.194430:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758336.194433:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758336.194435:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758336.194437:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758336.194440:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758336.194443:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758336.194446:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758336.194451:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737856:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758336.194456:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758336.194459:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758336.194461:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758336.194463:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758336.194468:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194489:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758336.194491:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:1.0:1787758336.194498:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758336.194518:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758336.194526:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758336.194529:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758336.194531:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194533:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758336.194537:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194542:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306063374 00000100:00000001:1.0:1787758336.194546:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306064065 00000100:00000001:1.0:1787758336.194549:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306063374 00000100:00000001:1.0:1787758336.194552:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306064065 00000100:00000001:1.0:1787758336.194555:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758336.194557:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194559:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758336.194565:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758336/real 1787758336] req@ffff8ffb47a38000 x1874588659737856/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758336.194586:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758336.194588:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758336.194591:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758336.194593:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758336.194595:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194597:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758336.194599:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758336.194603:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659737856/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758336.194612:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194615:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758336.194619:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194621:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758336.194622:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194625:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758336.194638:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758336.194642:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.194646:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758336.194648:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758336.194652:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737856:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758336.194658:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758336.194659:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758336.194661:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758336.194689:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758336.194692:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758336.194694:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758336.194698:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737984:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758336.194703:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758336.194705:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758336.194708:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758336.194710:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758336.194714:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194721:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758336.194723:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787758336.194728:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758336.194744:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758336.194750:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758336.194753:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758336.194755:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194757:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758336.194761:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194765:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306063371 00000100:00000001:1.0:1787758336.194768:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306064065 00000100:00000001:1.0:1787758336.194771:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306063371 00000100:00000001:1.0:1787758336.194774:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306064065 00000100:00000001:1.0:1787758336.194777:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758336.194779:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194781:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758336.194786:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758336/real 1787758336] req@ffff8ffb47a3b480 x1874588659737984/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758336.194796:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758336.194797:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758336.194800:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758336.194802:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758336.194804:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194806:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758336.194808:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758336.194813:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659737984/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758336.194821:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194824:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758336.194828:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194830:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758336.194832:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194834:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758336.194846:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758336.194849:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.194853:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758336.194856:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758336.194860:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659737984:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758336.194864:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758336.194865:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758336.194867:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758336.194869:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758336.194871:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758336.194874:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758336.194877:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758336.194882:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758336.194883:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758336.194886:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758336.194888:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758336.194891:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194895:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758336.194897:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787758336.194901:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758336.194912:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758336.194917:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758336.194919:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758336.194921:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194923:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758336.194926:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194930:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306063371 00000100:00000001:1.0:1787758336.194933:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306064065 00000100:00000001:1.0:1787758336.194936:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306063371 00000100:00000001:1.0:1787758336.194939:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306064065 00000100:00000001:1.0:1787758336.194942:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758336.194944:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.194946:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758336.194951:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758336/real 1787758336] req@ffff8ffb47a39180 x1874588659738112/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758336.194959:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758336.194961:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758336.194963:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758336.194965:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758336.194967:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194969:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758336.194970:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758336.194975:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659738112/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758336.194983:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194986:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758336.194989:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194991:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758336.194992:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.194994:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758336.195006:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758336.195009:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.195013:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758336.195015:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758336.195019:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738112:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758336.195024:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.195026:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758336.195028:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758336.195031:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758336.195033:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758336.195035:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758336.195037:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758336.195040:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758336.195041:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758336.195044:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758336.195046:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.195048:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758336.195050:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758336.195051:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758336.195054:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758336.195056:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758336.195057:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758336.195059:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758336.195060:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758336.195063:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758336.195065:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.195067:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758336.195069:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758336.195071:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758336.195072:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758336.195074:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758336.195075:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758336.195076:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758336.195078:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758336.195081:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758336.195082:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.195084:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758336.195089:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758336.195090:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758336.195093:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758336.195095:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758336.769231:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758336.769237:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758336.769239:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758336.769243:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758336.769246:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758336.769249:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758336.769250:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758336.769252:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758336.769254:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758336.961274:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758336.961280:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758336.961285:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758336.961287:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758336.961305:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758336.961307:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758336.961309:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758337.793251:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758337.793258:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758337.793260:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758337.793264:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758337.793267:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758337.793270:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758337.793271:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758337.793273:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758337.793275:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758337.985295:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758337.985302:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758337.985304:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758337.985307:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758337.985310:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758337.985312:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758337.985313:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758337.985315:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758337.985318:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758337.985319:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758337.985360:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758337.985364:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758337.985366:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758338.817305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758338.817315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758338.817317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758338.817323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758338.817327:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758338.817331:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758338.817333:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758338.817335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758338.817337:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758339.009270:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758339.009276:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758339.009281:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758339.009283:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758339.009332:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758339.009336:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758339.009339:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758339.841333:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758339.841340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758339.841342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758339.841347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758339.841350:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758339.841353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758339.841356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758339.841358:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758339.841360:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758340.033313:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758340.033339:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758340.033350:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758340.033354:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758340.033483:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758340.033489:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758340.033494:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758340.865276:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758340.865282:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758340.865284:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758340.865288:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758340.865291:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758340.865294:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758340.865296:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758340.865298:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758340.865300:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758341.057289:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758341.057295:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758341.057299:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758341.057300:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758341.057318:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758341.057320:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758341.057322:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758341.313282:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758341.313292:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758341.313294:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758341.313299:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758341.313303:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758341.313310:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11396 00000100:00080000:2.0:1787758341.313319:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758341.313323:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758341.313325:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758341.313329:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758341.313332:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758341.313335:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758341.313338:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758341.313341:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758341.313342:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758341.313347:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758341.313350:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758341.313353:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758341.313355:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758341.313359:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313361:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758341.313364:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758341.313375:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758341.313377:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313380:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758341.313383:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758341.313386:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313398:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758341.313407:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659738240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758341.313419:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758341.313423:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313441:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758341.313444:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758341.313450:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758341.313452:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758341.313453:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758341.313457:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758341.313460:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11396 00000100:00080000:2.0:1787758341.313464:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758341.313467:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758341.313468:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758341.313471:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758341.313473:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758341.313476:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758341.313479:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758341.313481:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758341.313484:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758341.313484:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758341.313489:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:2.0:1787758341.313491:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758341.313493:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758341.313496:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313497:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787758341.313499:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758341.313501:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758341.313506:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758341.313509:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758341.313510:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.313515:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758341.313515:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758341.313521:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758341.313522:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787758341.313526:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313531:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758341.313533:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659738240:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787758341.313535:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758341.313538:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313543:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787758341.313543:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758341.313548:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758341.313550:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313557:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.313559:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758341.313563:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758341.313565:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659738368/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758341.313566:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758341.313597:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313600:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758341.313604:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313608:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758341.313611:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758341.313614:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758341.313617:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758341.313617:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758341.313622:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00080000:2.0:1787758341.313623:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758341.313628:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758341.313631:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:2.0:1787758341.313632:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11396 00000100:00080000:2.0:1787758341.313638:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758341.313641:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758341.313643:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758341.313645:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758341.313647:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758341.313650:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758341.313653:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758341.313655:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758341.313657:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758341.313660:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758341.313663:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758341.313665:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758341.313668:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758341.313671:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758341.313674:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758341.313675:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758341.313680:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758341.313684:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313686:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313688:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787758341.313692:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758341.313694:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787758341.313695:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313697:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313700:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758341.313701:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:2.0:1787758341.313705:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758341.313708:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758341.313708:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313714:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306069185 00000100:00000001:2.0:1787758341.313715:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313722:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00100000:2.0:1787758341.313723:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659738496/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758341.313728:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306069185 00000100:00000001:1.0:1787758341.313734:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758341.313739:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758341.313740:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758341.313743:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313744:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313746:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758341.313750:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758341.313755:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758341/real 1787758341] req@ffff8ffb47a3a300 x1874588659738240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758341.313755:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758341.313766:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758341.313770:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000400:00000001:1.0:1787758341.313773:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:2.0:1787758341.313774:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000400:00000001:1.0:1787758341.313776:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758341.313780:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758341.313781:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758341.313785:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00080000:2.0:1787758341.313786:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11396 00000100:00000001:1.0:1787758341.313787:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.313793:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:2.0:1787758341.313794:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758341.313795:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758341.313801:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00100000:1.0:1787758341.313804:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659738240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758341.313805:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758341.313815:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.313817:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758341.313819:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787758341.313821:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758341.313822:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758341.313825:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1787758341.313826:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758341.313828:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758341.313831:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:2.0:1787758341.313832:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:1.0:1787758341.313833:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000020:00000001:2.0:1787758341.313839:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:1.0:1787758341.313840:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758341.313847:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758341.313848:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1787758341.313853:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787758341.313854:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659738240:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758341.313862:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:2.0:1787758341.313864:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758341.313865:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758341.313868:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758341.313871:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:2.0:1787758341.313872:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.313874:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 02000000:00000001:2.0:1787758341.313876:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000020:00000001:1.0:1787758341.313877:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758341.313880:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758341.313881:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758341.313884:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758341.313886:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.313887:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758341.313891:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758341.313893:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000008:00100000:2.0:1787758341.313895:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758341.313896:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.313900:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:2.0:1787758341.313902:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313906:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313907:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758341.313910:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758341.313911:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758341.313913:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:2.0:1787758341.313915:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313916:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758341.313920:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758341.313921:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758341.313923:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:2.0:1787758341.313925:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758341.313926:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.313929:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758341.313930:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.313931:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758341.313935:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758341.313938:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758341.313939:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738368:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758341.313944:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758341.313945:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758341.313948:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758341.313949:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659738624/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758341.313950:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758341.313954:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313961:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758341.313964:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00000001:2.0:1787758341.313964:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758341.313969:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313970:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758341.313976:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758341.313984:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758341.313988:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758341.313991:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758341.313992:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313993:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758341.313995:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.313999:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306068493 00000100:00000001:1.0:1787758341.314002:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306069185 00000100:00000001:1.0:1787758341.314006:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306068493 00000100:00000001:1.0:1787758341.314008:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306069185 00000100:00000001:1.0:1787758341.314010:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758341.314013:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314015:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758341.314019:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758341/real 1787758341] req@ffff8ffb47a38700 x1874588659738368/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758341.314029:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758341.314031:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758341.314033:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758341.314035:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758341.314037:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314038:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758341.314040:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758341.314044:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659738368/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758341.314058:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314060:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758341.314061:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314063:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758341.314064:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314065:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758341.314075:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758341.314077:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758341.314079:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758341.314081:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758341.314083:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738368:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758341.314086:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758341.314087:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758341.314088:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.314090:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758341.314091:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758341.314093:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758341.314095:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738496:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758341.314097:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758341.314098:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758341.314099:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.314100:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758341.314102:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314107:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758341.314108:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787758341.314111:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758341.314120:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758341.314123:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758341.314125:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758341.314126:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314127:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758341.314129:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314131:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306068490 00000100:00000001:1.0:1787758341.314133:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306069185 00000100:00000001:1.0:1787758341.314134:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306068490 00000100:00000001:1.0:1787758341.314136:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306069185 00000100:00000001:1.0:1787758341.314137:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758341.314139:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314140:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758341.314142:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758341/real 1787758341] req@ffff8ffb47a39180 x1874588659738496/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758341.314147:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758341.314147:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758341.314149:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758341.314150:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758341.314151:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314152:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758341.314153:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758341.314155:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659738496/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758341.314160:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314161:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758341.314163:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314164:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758341.314165:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314166:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758341.314174:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758341.314176:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758341.314178:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758341.314179:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758341.314181:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738496:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758341.314208:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314209:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758341.314210:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758341.314211:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758341.314213:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758341.314214:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758341.314215:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758341.314217:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758341.314218:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758341.314220:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758341.314221:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314222:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758341.314223:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758341.314224:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758341.314226:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758341.314226:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758341.314227:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758341.314228:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758341.314229:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758341.314231:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758341.314232:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314233:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314236:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758341.314237:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314238:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758341.314239:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758341.314240:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758341.314241:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758341.314242:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.314244:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758341.314245:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758341.314246:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758341.314249:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758341.314251:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758341.314252:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758341.314253:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758341.314254:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758341.314257:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314260:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758341.314261:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787758341.314264:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758341.314271:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758341.314282:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758341.314285:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758341.314286:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314288:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758341.314291:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314294:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306068491 00000100:00000001:1.0:1787758341.314297:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306069185 00000100:00000001:1.0:1787758341.314299:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306068491 00000100:00000001:1.0:1787758341.314302:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306069185 00000100:00000001:1.0:1787758341.314304:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758341.314306:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314308:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758341.314313:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758341/real 1787758341] req@ffff8ffb47a3b480 x1874588659738624/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758341.314322:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758341.314324:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758341.314326:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758341.314328:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758341.314330:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314331:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758341.314333:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758341.314336:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659738624/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758341.314341:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314343:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758341.314345:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314346:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758341.314347:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314348:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758341.314358:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758341.314359:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758341.314362:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758341.314363:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758341.314365:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738624:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758341.314367:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314369:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758341.314369:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758341.314370:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758341.314372:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758341.314373:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758341.314373:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758341.314375:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758341.314376:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758341.314377:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758341.314378:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314380:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758341.314382:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758341.314383:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758341.314384:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758341.314385:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758341.889232:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758341.889239:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758341.889242:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758341.889247:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758341.889251:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758341.889256:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758341.889258:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758341.889262:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758341.889265:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758342.081248:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758342.081254:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758342.081260:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758342.081270:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758342.081292:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758342.081295:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758342.081298:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758342.913295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758342.913302:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758342.913304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758342.913310:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758342.913313:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758342.913317:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758342.913320:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758342.913330:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758342.913333:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758343.105303:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758343.105309:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758343.105314:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758343.105315:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758343.105349:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758343.105355:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758343.105359:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758343.937235:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758343.937241:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758343.937242:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758343.937246:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758343.937249:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758343.937252:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758343.937253:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758343.937255:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758343.937257:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758344.129269:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758344.129274:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758344.129279:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758344.129280:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758344.129308:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758344.129312:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758344.129315:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758344.961311:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758344.961317:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758344.961319:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758344.961323:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758344.961326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758344.961329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758344.961331:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758344.961333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758344.961335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758345.153249:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758345.153257:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758345.153264:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758345.153268:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758345.153270:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758345.153273:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758345.153276:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758345.153279:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758345.153281:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758345.153282:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758345.153300:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758345.153302:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758345.153304:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758345.985248:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758345.985257:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758345.985259:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758345.985264:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758345.985267:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758345.985270:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758345.985272:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758345.985274:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758345.985276:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758346.177290:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758346.177296:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758346.177302:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758346.177303:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758346.177348:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758346.177351:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758346.177354:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758346.433298:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758346.433305:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758346.433309:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758346.433313:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758346.433317:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758346.433326:0:178062:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11401 00000100:00080000:1.0:1787758346.433337:0:178062:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758346.433342:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758346.433344:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758346.433349:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758346.433353:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758346.433355:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758346.433357:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758346.433359:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758346.433360:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758346.433364:0:178062:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758346.433365:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758346.433367:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758346.433370:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758346.433373:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433374:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758346.433376:0:178062:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758346.433385:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758346.433386:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433388:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758346.433391:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758346.433394:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433403:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758346.433411:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659738752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758346.433420:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758346.433424:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433466:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758346.433471:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758346.433475:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758346.433478:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758346.433480:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758346.433502:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:3.0:1787758346.433503:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:1.0:1787758346.433507:0:178062:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11401 00000100:00000001:3.0:1787758346.433509:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:1.0:1787758346.433513:0:178062:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758346.433513:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758346.433516:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758346.433516:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758346.433519:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:3.0:1787758346.433519:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758346.433522:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758346.433524:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:3.0:1787758346.433525:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:1.0:1787758346.433527:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 02000000:00000001:3.0:1787758346.433528:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:1.0:1787758346.433530:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758346.433532:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:3.0:1787758346.433532:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433535:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:3.0:1787758346.433538:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659738752:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787758346.433540:0:178062:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758346.433543:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.433544:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758346.433545:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758346.433551:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758346.433553:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433555:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758346.433560:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758346.433564:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433567:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758346.433572:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758346.433573:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433575:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758346.433577:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758346.433579:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433586:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758346.433594:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659738880/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758346.433603:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758346.433606:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433609:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758346.433612:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758346.433616:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758346.433618:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758346.433620:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:3.0:1787758346.433620:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758346.433623:0:178062:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:3.0:1787758346.433624:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758346.433626:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11401 02000000:00000001:3.0:1787758346.433627:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:1.0:1787758346.433631:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:3.0:1787758346.433632:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433633:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758346.433635:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758346.433637:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758346.433639:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758346.433641:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758346.433643:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758346.433645:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758346.433647:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758346.433651:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758346.433654:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758346.433656:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758346.433659:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758346.433662:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433664:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758346.433667:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758346.433670:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433672:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758346.433677:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758346.433679:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433681:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758346.433682:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758346.433684:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433689:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758346.433695:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659739008/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758346.433713:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758346.433715:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433719:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758346.433722:0:178062:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758346.433726:0:178062:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758346.433728:0:178062:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758346.433730:0:178062:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758346.433733:0:178062:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758346.433736:0:178062:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11401 00000100:00080000:1.0:1787758346.433740:0:178062:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758346.433743:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758346.433745:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758346.433747:0:178062:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758346.433749:0:178062:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758346.433751:0:178062:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758346.433754:0:178062:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758346.433756:0:178062:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758346.433758:0:178062:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758346.433761:0:178062:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758346.433762:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758346.433764:0:178062:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.433765:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 02000000:00000001:1.0:1787758346.433782:0:178062:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758346.433785:0:178062:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758346.433786:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:1.0:1787758346.433788:0:178062:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433790:0:178062:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758346.433793:0:178062:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758346.433795:0:178062:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433797:0:178062:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758346.433801:0:178062:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758346.433803:0:178062:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433805:0:178062:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758346.433806:0:178062:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758346.433809:0:178062:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433814:0:178062:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758346.433820:0:178062:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659739136/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758346.433824:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758346.433829:0:178062:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758346.433831:0:178062:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758346.433849:0:178062:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.433850:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758346.433904:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758346.433908:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.433911:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758346.433932:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.433937:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306074305 00000100:00000001:3.0:1787758346.433944:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758346.433948:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306074305 00000100:00000001:3.0:1787758346.433951:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758346.433954:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758346.433956:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.433958:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758346.433962:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758346/real 1787758346] req@ffff8ffb4a534a80 x1874588659738752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758346.433970:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758346.433972:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758346.433975:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758346.433978:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758346.433979:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.433981:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758346.433982:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758346.433985:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659738752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758346.433990:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.433993:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758346.433995:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.433996:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758346.433997:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.433999:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758346.434000:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758346.434002:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758346.434004:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758346.434006:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758346.434008:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659738752:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758346.434012:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434030:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758346.434031:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758346.434033:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758346.434035:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758346.434037:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758346.434039:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758346.434042:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758346.434044:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758346.434046:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758346.434047:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434048:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434054:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758346.434056:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434057:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758346.434059:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758346.434060:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758346.434061:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758346.434062:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758346.434064:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758346.434066:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758346.434067:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758346.434070:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738880:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758346.434073:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758346.434074:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758346.434075:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758346.434077:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758346.434079:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434084:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758346.434085:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:3.0:1787758346.434089:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758346.434100:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758346.434104:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758346.434105:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758346.434107:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434108:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758346.434110:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434113:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306073613 00000100:00000001:3.0:1787758346.434116:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306074305 00000100:00000001:3.0:1787758346.434118:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306073613 00000100:00000001:3.0:1787758346.434119:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306074305 00000100:00000001:3.0:1787758346.434121:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758346.434123:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434124:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758346.434127:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758346/real 1787758346] req@ffff8ffb4a536300 x1874588659738880/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758346.434132:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758346.434133:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758346.434135:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758346.434136:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758346.434137:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434139:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758346.434140:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758346.434143:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659738880/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758346.434149:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434152:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758346.434155:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434157:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758346.434158:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434160:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758346.434175:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758346.434179:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758346.434199:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758346.434202:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758346.434207:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659738880:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758346.434213:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758346.434216:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758346.434218:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758346.434222:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758346.434226:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758346.434278:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758346.434283:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739008:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758346.434287:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758346.434289:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758346.434291:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758346.434294:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758346.434297:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434305:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758346.434307:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:3.0:1787758346.434311:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758346.434323:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758346.434327:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758346.434329:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758346.434330:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434332:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758346.434334:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434336:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306073611 00000100:00000001:3.0:1787758346.434338:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306074305 00000100:00000001:3.0:1787758346.434340:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306073611 00000100:00000001:3.0:1787758346.434342:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306074305 00000100:00000001:3.0:1787758346.434343:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758346.434344:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434346:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758346.434348:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758346/real 1787758346] req@ffff8ffb4a537b80 x1874588659739008/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758346.434354:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758346.434354:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758346.434356:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758346.434357:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758346.434358:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434359:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758346.434360:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758346.434363:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659739008/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758346.434367:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434368:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758346.434370:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434371:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758346.434372:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434373:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758346.434381:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758346.434383:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758346.434385:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758346.434387:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758346.434389:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739008:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758346.434392:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758346.434393:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758346.434394:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758346.434395:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758346.434396:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758346.434398:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758346.434399:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758346.434402:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758346.434403:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758346.434404:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758346.434405:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758346.434407:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434410:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758346.434411:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:3.0:1787758346.434413:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:3.0:1787758346.434419:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758346.434422:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758346.434423:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758346.434424:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434426:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758346.434427:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434429:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306073611 00000100:00000001:3.0:1787758346.434431:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306074305 00000100:00000001:3.0:1787758346.434433:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306073611 00000100:00000001:3.0:1787758346.434434:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306074305 00000100:00000001:3.0:1787758346.434436:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758346.434437:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434438:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:3.0:1787758346.434441:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758346/real 1787758346] req@ffff8ffb4a537800 x1874588659739136/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:3.0:1787758346.434446:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1787758346.434446:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758346.434448:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758346.434449:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758346.434450:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434451:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758346.434452:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758346.434454:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659739136/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758346.434459:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434460:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758346.434461:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434462:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758346.434463:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434464:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758346.434473:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758346.434474:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758346.434476:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758346.434477:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758346.434479:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739136:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758346.434481:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434513:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758346.434515:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758346.434517:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758346.434519:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758346.434520:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758346.434521:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758346.434523:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758346.434524:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758346.434526:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758346.434527:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434529:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758346.434530:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758346.434531:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758346.434532:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758346.434533:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758346.434534:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758346.434535:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758346.434536:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758346.434537:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758346.434538:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758346.434540:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758346.434541:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758346.434542:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1787758346.434543:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:3.0:1787758346.434543:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:3.0:1787758346.434545:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:3.0:1787758346.434546:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758346.434547:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:3.0:1787758346.434548:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434566:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758346.434569:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758346.434569:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758346.434571:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758346.434572:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758347.009266:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758347.009272:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758347.009274:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758347.009278:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758347.009281:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758347.009285:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758347.009287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758347.009289:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758347.009291:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758347.201250:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758347.201257:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758347.201259:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758347.201262:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758347.201266:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758347.201268:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758347.201270:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758347.201272:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758347.201276:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758347.201277:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758347.201310:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758347.201314:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758347.201316:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758348.033242:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758348.033249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758348.033250:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758348.033255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758348.033258:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758348.033261:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758348.033263:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758348.033265:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758348.033267:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758348.225262:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758348.225267:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758348.225272:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758348.225274:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758348.225321:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758348.225324:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758348.225327:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758349.057298:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758349.057310:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758349.057313:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758349.057320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758349.057323:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758349.057328:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758349.057330:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758349.057333:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758349.057335:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758349.249296:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758349.249302:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758349.249308:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758349.249309:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758349.249329:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758349.249331:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758349.249334:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758350.081245:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758350.081254:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758350.081257:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758350.081262:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758350.081266:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758350.081269:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758350.081271:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758350.081274:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758350.081277:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758350.273276:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758350.273282:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758350.273286:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758350.273287:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758350.273329:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758350.273332:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758350.273335:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758351.105260:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758351.105266:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758351.105268:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758351.105272:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758351.105275:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758351.105278:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758351.105280:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758351.105282:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758351.105284:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758351.297322:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758351.297331:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758351.297339:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758351.297341:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758351.297420:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758351.297427:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758351.297432:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758351.553263:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758351.553270:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758351.553272:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758351.553275:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758351.553278:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758351.553284:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11406 00000100:00080000:3.0:1787758351.553292:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758351.553295:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758351.553296:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758351.553299:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758351.553301:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758351.553304:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758351.553305:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758351.553307:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758351.553308:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758351.553311:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758351.553313:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758351.553315:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758351.553318:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758351.553320:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553321:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758351.553323:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758351.553332:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758351.553333:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553336:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758351.553338:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758351.553341:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553351:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758351.553358:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659739264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758351.553366:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758351.553372:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553389:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758351.553402:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758351.553407:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758351.553409:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758351.553411:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758351.553415:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758351.553415:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758351.553420:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00080000:3.0:1787758351.553420:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11406 00000100:00000001:1.0:1787758351.553424:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758351.553426:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787758351.553427:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758351.553428:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553435:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758351.553436:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758351.553438:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758351.553439:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758351.553443:0:87148:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:1.0:1787758351.553444:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758351.553446:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758351.553449:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00100000:1.0:1787758351.553451:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659739264:192.168.201.118@tcp:250:kworker.0 00000020:00000001:3.0:1787758351.553452:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758351.553457:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758351.553459:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758351.553459:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758351.553463:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787758351.553465:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787758351.553467:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553468:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.553469:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758351.553471:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758351.553475:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758351.553476:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758351.553480:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553481:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758351.553485:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758351.553487:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553489:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553490:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758351.553493:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 02000000:00000001:3.0:1787758351.553493:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758351.553496:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553497:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758351.553500:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758351.553503:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758351.553505:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553510:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758351.553516:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659739392/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758351.553522:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758351.553524:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553527:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758351.553529:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758351.553532:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787758351.553534:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758351.553535:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758351.553538:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758351.553541:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758351.553544:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11406 00000100:00000001:1.0:1787758351.553546:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:3.0:1787758351.553548:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758351.553551:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758351.553552:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758351.553554:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553555:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758351.553559:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000020:00000001:3.0:1787758351.553560:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758351.553562:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758351.553564:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758351.553565:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758351.553566:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758351.553570:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306079425 00000100:00000001:3.0:1787758351.553570:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758351.553571:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758351.553577:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758351.553578:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758351.553579:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758351.553581:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787758351.553582:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758351.553587:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553588:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758351.553589:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306079425 00000008:00100000:3.0:1787758351.553590:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758351.553594:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553595:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758351.553597:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553599:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758351.553602:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758351.553603:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758351.553605:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553607:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758351.553609:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758351.553611:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758351.553613:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758351.553614:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758351/real 1787758351] req@ffff8ffb45c31c00 x1874588659739264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758351.553617:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758351.553627:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659739520/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:1.0:1787758351.553629:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758351.553631:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758351.553635:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758351.553637:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758351.553639:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758351.553641:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553642:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553646:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758351.553646:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553649:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758351.553650:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758351.553654:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758351.553656:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659739264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758351.553656:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758351.553659:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758351.553661:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758351.553663:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11406 00000100:00080000:3.0:1787758351.553665:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758351.553669:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758351.553671:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758351.553676:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758351.553679:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787758351.553680:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758351.553683:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758351.553685:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787758351.553686:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758351.553690:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758351.553691:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758351.553694:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758351.553695:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758351.553697:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758351.553700:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758351.553701:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758351.553705:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758351.553705:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758351.553708:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00080000:1.0:1787758351.553710:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:3.0:1787758351.553711:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758351.553714:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758351.553715:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758351.553717:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758351.553718:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000008:00100000:3.0:1787758351.553721:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758351.553724:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659739264:192.168.201.118@tcp:250:kworker.0 00000008:00000001:3.0:1787758351.553725:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553731:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553733:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553735:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787758351.553737:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758351.553739:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758351.553740:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758351.553741:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758351.553741:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758351.553742:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.553746:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:3.0:1787758351.553747:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.553751:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00100000:3.0:1787758351.553754:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659739648/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000001:1.0:1787758351.553755:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758351.553758:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758351.553762:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:3.0:1787758351.553764:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758351.553765:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758351.553767:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553769:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758351.553771:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758351.553771:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553776:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553782:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758351.553784:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553785:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758351.553787:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758351.553788:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758351.553789:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758351.553795:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.553796:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758351.553798:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758351.553799:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758351.553802:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739392:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758351.553805:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758351.553806:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758351.553807:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.553808:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758351.553810:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553814:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758351.553815:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 00000100:00000001:1.0:1787758351.553818:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758351.553827:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758351.553830:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758351.553831:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758351.553832:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553833:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758351.553835:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553838:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306078733 00000100:00000001:1.0:1787758351.553840:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306079425 00000100:00000001:1.0:1787758351.553842:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306078733 00000100:00000001:1.0:1787758351.553843:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306079425 00000100:00000001:1.0:1787758351.553845:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758351.553846:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553847:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758351.553851:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758351/real 1787758351] req@ffff8ffb45c30e00 x1874588659739392/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758351.553856:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758351.553857:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758351.553859:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758351.553860:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758351.553861:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553862:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758351.553863:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758351.553866:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659739392/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758351.553870:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553871:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758351.553873:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553874:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758351.553875:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553876:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758351.553885:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758351.553887:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758351.553889:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758351.553891:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758351.553893:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739392:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758351.553895:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758351.553896:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758351.553897:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.553898:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758351.553899:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758351.553901:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758351.553903:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739520:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758351.553907:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758351.553908:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758351.553909:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.553910:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758351.553912:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553915:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758351.553916:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787758351.553919:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758351.553925:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758351.553927:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758351.553929:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758351.553930:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553931:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758351.553932:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553934:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306078730 00000100:00000001:1.0:1787758351.553936:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306079425 00000100:00000001:1.0:1787758351.553937:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306078730 00000100:00000001:1.0:1787758351.553939:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306079425 00000100:00000001:1.0:1787758351.553941:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758351.553942:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.553943:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758351.553945:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758351/real 1787758351] req@ffff8ffb45c33b80 x1874588659739520/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758351.553950:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758351.553951:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758351.553952:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758351.553953:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758351.553954:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553955:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758351.553955:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758351.553957:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659739520/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758351.553966:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553968:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758351.553969:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553970:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758351.553971:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.553972:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758351.553980:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758351.553981:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758351.553983:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758351.553984:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758351.553986:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739520:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758351.553989:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758351.553990:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758351.553991:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.553992:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758351.553993:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758351.553995:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758351.553996:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758351.553998:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758351.553999:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758351.554000:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758351.554001:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758351.554003:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.554005:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758351.554006:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787758351.554009:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758351.554014:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758351.554017:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758351.554018:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758351.554019:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.554020:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758351.554022:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.554024:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306078730 00000100:00000001:1.0:1787758351.554026:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306079425 00000100:00000001:1.0:1787758351.554027:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306078730 00000100:00000001:1.0:1787758351.554029:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306079425 00000100:00000001:1.0:1787758351.554030:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758351.554031:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.554032:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758351.554034:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758351/real 1787758351] req@ffff8ffb45c30a80 x1874588659739648/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758351.554039:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758351.554039:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758351.554041:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758351.554041:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758351.554042:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554043:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758351.554044:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758351.554046:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659739648/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758351.554051:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554052:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758351.554054:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554055:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758351.554056:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554056:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758351.554064:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758351.554066:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758351.554068:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758351.554069:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758351.554071:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739648:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758351.554074:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554075:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758351.554076:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758351.554077:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758351.554078:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758351.554079:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758351.554080:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758351.554081:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758351.554082:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758351.554083:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758351.554084:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554085:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758351.554086:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758351.554087:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758351.554089:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758351.554090:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758351.554090:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758351.554091:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758351.554092:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758351.554093:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758351.554094:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554094:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758351.554095:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758351.554096:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758351.554103:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758351.554104:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758351.554105:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758351.554108:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758351.554109:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758351.554111:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758351.554112:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554114:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758351.554117:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758351.554119:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758351.554121:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758351.554122:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758352.129255:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758352.129262:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758352.129264:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758352.129268:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758352.129271:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758352.129274:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758352.129276:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758352.129278:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758352.129280:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758352.321291:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758352.321298:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758352.321304:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758352.321305:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758352.321335:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758352.321338:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758352.321341:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758353.153264:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758353.153270:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758353.153272:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758353.153276:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758353.153279:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758353.153282:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758353.153284:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758353.153285:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758353.153287:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758353.345258:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758353.345263:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758353.345268:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758353.345270:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758353.345286:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758353.345287:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758353.345289:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758354.177234:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758354.177240:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758354.177242:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758354.177245:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758354.177248:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758354.177251:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758354.177253:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758354.177255:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758354.177257:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758354.369281:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758354.369287:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758354.369291:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758354.369292:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758354.369317:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758354.369320:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758354.369323:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758355.201243:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758355.201249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758355.201250:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758355.201254:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758355.201257:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758355.201260:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758355.201262:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758355.201264:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758355.201267:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758355.393303:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758355.393313:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758355.393321:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758355.393347:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758355.393352:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758355.393358:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758355.393366:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758355.393370:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758355.393373:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758355.393378:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758355.393474:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758355.393481:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758355.393485:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758356.225295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758356.225303:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758356.225306:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758356.225337:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758356.225342:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758356.225347:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758356.225350:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758356.225354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758356.225357:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758356.417363:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758356.417372:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758356.417381:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758356.417384:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758356.417472:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758356.417478:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758356.417483:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758356.673398:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758356.673411:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758356.673414:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758356.673418:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758356.673423:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758356.673432:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11411 00000100:00080000:2.0:1787758356.673447:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758356.673450:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758356.673452:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758356.673456:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758356.673459:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758356.673462:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758356.673464:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758356.673466:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758356.673467:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758356.673471:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758356.673473:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758356.673475:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758356.673480:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758356.673483:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673485:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758356.673487:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758356.673504:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758356.673505:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673509:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758356.673512:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758356.673517:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673533:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758356.673547:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659739776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758356.673560:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758356.673566:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673588:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758356.673591:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758356.673594:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758356.673596:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758356.673597:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758356.673600:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758356.673602:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11411 00000100:00080000:2.0:1787758356.673606:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758356.673608:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758356.673609:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758356.673610:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758356.673612:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758356.673613:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758356.673615:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758356.673616:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758356.673623:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758356.673626:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758356.673628:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758356.673633:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758356.673639:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758356.673640:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673642:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758356.673647:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758356.673651:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673652:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758356.673656:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758356.673658:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673659:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758356.673662:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758356.673665:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673671:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758356.673676:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659739904/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758356.673683:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758356.673686:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.673688:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758356.673689:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758356.673693:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758356.673699:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758356.673700:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758356.673702:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758356.673705:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758356.673706:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758356.673708:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758356.673712:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:2.0:1787758356.673713:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11411 00000100:00000001:1.0:1787758356.673719:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758356.673721:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758356.673730:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758356.673733:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787758356.673733:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758356.673743:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:1.0:1787758356.673744:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1787758356.673746:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758356.673755:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758356.673756:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758356.673774:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758356.673780:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758356.673783:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:1.0:1787758356.673785:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659739776:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787758356.673791:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758356.673801:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.673802:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758356.673815:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758356.673818:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758356.673821:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673822:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787758356.673824:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00100000:2.0:1787758356.673826:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758356.673829:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787758356.673831:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758356.673838:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758356.673839:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758356.673851:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758356.673852:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673854:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758356.673856:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758356.673856:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758356.673862:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673869:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758356.673877:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659740032/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758356.673884:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758356.673885:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758356.673888:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:2.0:1787758356.673890:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.673901:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758356.673902:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758356.673912:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758356.673921:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758356.673924:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758356.673925:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758356.673928:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758356.673932:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11411 00000100:00080000:2.0:1787758356.673936:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758356.673938:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758356.673939:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758356.673942:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758356.673944:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758356.673946:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758356.673949:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758356.673950:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758356.673951:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758356.673954:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758356.673956:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758356.673958:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758356.673960:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758356.673961:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673963:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758356.673965:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758356.673978:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:2.0:1787758356.673978:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673981:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758356.673988:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758356.673991:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.673994:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758356.673995:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758356.673997:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.674003:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674012:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:2.0:1787758356.674012:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659740160/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758356.674026:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758356.674027:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758356.674033:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674035:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758356.674038:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674041:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758356.674053:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674060:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306084545 00000100:00000001:1.0:1787758356.674075:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758356.674088:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306084545 00000100:00000001:1.0:1787758356.674099:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758356.674105:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758356.674111:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674116:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758356.674130:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758356/real 1787758356] req@ffff8ffb47a38000 x1874588659739776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758356.674167:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758356.674170:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758356.674176:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758356.674179:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758356.674224:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674229:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758356.674232:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758356.674240:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659739776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758356.674251:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674255:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758356.674259:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674261:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758356.674262:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674265:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758356.674268:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758356.674272:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758356.674277:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758356.674280:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758356.674285:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659739776:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758356.674291:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758356.674293:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758356.674295:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758356.674299:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758356.674302:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758356.674328:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758356.674333:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739904:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758356.674337:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758356.674340:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758356.674344:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758356.674346:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758356.674350:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674359:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758356.674361:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:1.0:1787758356.674367:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758356.674387:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758356.674394:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758356.674397:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758356.674399:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674401:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758356.674405:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674410:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306083854 00000100:00000001:1.0:1787758356.674414:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306084545 00000100:00000001:1.0:1787758356.674417:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306083854 00000100:00000001:1.0:1787758356.674420:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306084545 00000100:00000001:1.0:1787758356.674423:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758356.674427:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674430:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758356.674437:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758356/real 1787758356] req@ffff8ffb47a38380 x1874588659739904/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758356.674449:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758356.674451:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758356.674457:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758356.674461:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758356.674463:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674466:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758356.674468:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758356.674473:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659739904/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758356.674483:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674486:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758356.674489:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674491:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758356.674493:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674495:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758356.674511:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758356.674514:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758356.674520:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758356.674524:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758356.674528:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659739904:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758356.674535:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674539:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758356.674541:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758356.674545:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758356.674548:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758356.674551:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758356.674553:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758356.674558:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758356.674560:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758356.674565:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758356.674566:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674568:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758356.674570:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758356.674571:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758356.674574:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758356.674576:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758356.674577:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758356.674580:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758356.674582:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758356.674583:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758356.674585:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674587:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674596:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758356.674598:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674601:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758356.674603:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758356.674605:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758356.674607:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758356.674609:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758356.674612:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758356.674615:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758356.674617:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758356.674621:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740032:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758356.674625:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758356.674627:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758356.674629:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758356.674631:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758356.674634:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674640:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758356.674642:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:1.0:1787758356.674646:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758356.674660:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758356.674665:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758356.674668:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758356.674670:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674672:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758356.674675:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674679:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306083851 00000100:00000001:1.0:1787758356.674682:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306084545 00000100:00000001:1.0:1787758356.674685:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306083851 00000100:00000001:1.0:1787758356.674688:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306084545 00000100:00000001:1.0:1787758356.674691:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758356.674693:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674695:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758356.674700:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758356/real 1787758356] req@ffff8ffb47a39880 x1874588659740032/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758356.674710:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758356.674712:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758356.674714:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758356.674716:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758356.674718:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674721:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758356.674723:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758356.674727:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659740032/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758356.674736:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674738:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758356.674741:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674743:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758356.674744:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674746:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758356.674759:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758356.674761:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758356.674765:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758356.674768:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758356.674771:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740032:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758356.674776:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758356.674777:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758356.674779:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758356.674783:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758356.674785:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758356.674787:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758356.674791:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740160:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758356.674794:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758356.674796:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758356.674798:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758356.674800:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758356.674803:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674808:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758356.674809:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787758356.674814:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758356.674824:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758356.674829:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758356.674832:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758356.674833:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674835:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758356.674838:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674842:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306083851 00000100:00000001:1.0:1787758356.674846:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306084545 00000100:00000001:1.0:1787758356.674848:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306083851 00000100:00000001:1.0:1787758356.674851:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306084545 00000100:00000001:1.0:1787758356.674854:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758356.674857:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.674859:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758356.674863:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758356/real 1787758356] req@ffff8ffb47a3a680 x1874588659740160/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758356.674872:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758356.674873:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758356.674875:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758356.674877:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758356.674879:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674881:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758356.674882:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758356.674887:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659740160/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758356.674895:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674898:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758356.674901:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674903:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758356.674905:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674907:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758356.674919:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758356.674924:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758356.674931:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758356.674936:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758356.674942:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740160:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758356.674950:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674953:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758356.674956:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758356.674958:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758356.674963:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758356.674965:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758356.674968:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758356.674971:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758356.674974:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758356.674979:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758356.674982:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.674985:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758356.674987:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758356.674990:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758356.674993:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758356.674995:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758356.674998:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758356.675001:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758356.675004:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758356.675008:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758356.675011:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.675015:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758356.675021:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758356.675024:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758356.675028:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758356.675031:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758357.249266:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758357.249275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758357.249278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758357.249285:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758357.249290:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758357.249334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758357.249339:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758357.249342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758357.249346:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758357.441417:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758357.441430:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758357.441434:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758357.441439:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758357.441445:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758357.441448:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758357.441450:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758357.441453:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758357.441458:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758357.441460:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758357.441487:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758357.441490:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758357.441494:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758358.273405:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758358.273421:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758358.273438:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758358.273450:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758358.273457:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758358.273465:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758358.273470:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758358.273477:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758358.273482:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758358.465253:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758358.465258:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758358.465263:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758358.465264:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758358.465313:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758358.465316:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758358.465319:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758359.297333:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758359.297340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758359.297342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758359.297347:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758359.297351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758359.297354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758359.297357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758359.297359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758359.297361:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758359.489342:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758359.489350:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758359.489359:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758359.489362:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758359.489493:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758359.489499:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758359.489504:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758360.321373:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758360.321384:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758360.321388:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758360.321394:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758360.321399:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758360.321404:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758360.321408:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758360.321411:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758360.321415:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758360.513352:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758360.513361:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758360.513370:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758360.513372:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758360.513461:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758360.513467:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758360.513474:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758361.345325:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758361.345333:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758361.345336:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758361.345343:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758361.345348:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758361.345353:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758361.345357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758361.345360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758361.345364:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758361.537342:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758361.537351:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758361.537358:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758361.537361:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758361.537390:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758361.537394:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758361.537398:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758361.793423:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758361.793436:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758361.793439:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758361.793444:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758361.793450:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758361.793458:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11416 00000100:00080000:3.0:1787758361.793472:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758361.793477:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758361.793480:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758361.793485:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758361.793488:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758361.793492:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758361.793495:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758361.793498:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758361.793500:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758361.793506:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758361.793509:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758361.793513:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758361.793517:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758361.793521:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.793524:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758361.793527:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758361.793540:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758361.793542:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.793547:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758361.793550:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758361.793555:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.793570:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758361.793582:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659740288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758361.793605:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758361.793611:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.793635:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758361.793639:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758361.793644:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758361.793646:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758361.793648:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758361.793652:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758361.793655:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11416 00000100:00080000:3.0:1787758361.793659:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758361.793674:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758361.793676:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758361.793679:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758361.793681:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758361.793684:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758361.793687:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758361.793690:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758361.793693:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758361.793698:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758361.793702:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758361.793704:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758361.793711:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758361.793715:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.793716:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758361.793718:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758361.793727:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000008:00100000:3.0:1787758361.793728:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758361.793733:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000008:00000001:3.0:1787758361.793735:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.793741:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758361.793742:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.793749:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758361.793752:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758361.793757:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.793759:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758361.793761:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758361.793766:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758361.793767:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758361.793773:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.793775:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.793782:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758361.793789:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659740288:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787758361.793796:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659740416/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758361.793807:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758361.793815:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758361.793819:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758361.793820:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.793829:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.793838:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758361.793841:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.793847:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758361.793851:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758361.793861:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758361.793865:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758361.793868:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758361.793875:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758361.793877:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787758361.793882:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11416 00000100:00000001:1.0:1787758361.793886:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00080000:3.0:1787758361.793895:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758361.793898:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758361.793904:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758361.793910:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758361.793916:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758361.793919:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758361.793924:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758361.793929:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758361.793934:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758361.793938:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758361.793947:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758361.793951:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758361.793953:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758361.793958:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758361.793963:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.793969:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000800:00000100:1.0:1787758361.793971:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1787758361.793974:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758361.793981:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.793984:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758361.793991:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758361.793994:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.793997:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758361.793998:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758361.794004:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758361.794008:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794010:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758361.794017:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794019:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794028:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:3.0:1787758361.794030:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659740544/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758361.794040:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.794043:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758361.794050:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794053:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306089665 00000100:00000001:3.0:1787758361.794055:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758361.794062:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758361.794068:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758361.794070:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758361.794073:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758361.794077:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758361.794079:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:3.0:1787758361.794081:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11416 00000100:00080000:3.0:1787758361.794094:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758361.794097:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758361.794100:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306089665 00000100:00000001:3.0:1787758361.794100:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758361.794107:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758361.794112:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758361.794115:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758361.794118:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000020:00000001:3.0:1787758361.794119:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758361.794128:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758361.794131:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758361.794134:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787758361.794137:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758361.794148:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.794149:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758361.794157:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758361.794163:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 02000000:00000001:3.0:1787758361.794164:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758361.794172:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.794175:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758361.794178:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758361.794266:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.794271:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758361.794273:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758361/real 1787758361] req@ffff8ffb45c30700 x1874588659740288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758361.794281:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758361.794287:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758361.794296:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758361.794302:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758361.794309:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787758361.794313:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758361.794320:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758361.794321:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794330:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00100000:3.0:1787758361.794339:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659740672/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758361.794341:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758361.794349:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758361.794361:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758361.794362:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758361.794369:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794371:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758361.794375:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758361.794394:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659740288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758361.794416:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794422:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758361.794429:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794433:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758361.794436:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794440:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758361.794444:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758361.794450:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758361.794459:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758361.794464:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758361.794472:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659740288:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758361.794483:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794487:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758361.794491:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758361.794495:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758361.794501:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758361.794504:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758361.794526:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758361.794533:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758361.794536:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758361.794540:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758361.794543:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794548:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794559:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758361.794563:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794568:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758361.794571:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758361.794574:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758361.794577:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758361.794581:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.794585:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758361.794589:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758361.794593:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758361.794601:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740416:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758361.794609:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758361.794614:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758361.794617:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.794621:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758361.794627:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794640:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758361.794643:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787758361.794651:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758361.794680:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758361.794690:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758361.794694:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758361.794697:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794700:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758361.794706:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794713:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306088974 00000100:00000001:1.0:1787758361.794719:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306089665 00000100:00000001:1.0:1787758361.794723:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306088974 00000100:00000001:1.0:1787758361.794728:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306089665 00000100:00000001:1.0:1787758361.794732:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758361.794736:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794740:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758361.794749:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758361/real 1787758361] req@ffff8ffb45c32a00 x1874588659740416/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758361.794768:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758361.794771:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758361.794777:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758361.794780:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758361.794783:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794787:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758361.794789:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758361.794797:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659740416/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758361.794826:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794832:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758361.794837:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794840:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758361.794843:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.794847:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758361.794869:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758361.794875:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758361.794883:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758361.794888:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758361.794895:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740416:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758361.794904:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758361.794907:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758361.794910:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.794914:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758361.794918:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758361.794923:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758361.794929:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740544:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758361.794935:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758361.794938:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758361.794940:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.794943:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758361.794947:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794956:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758361.794958:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787758361.794963:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758361.794981:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758361.794988:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758361.794991:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758361.794993:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.794996:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758361.794999:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.795004:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306088971 00000100:00000001:1.0:1787758361.795008:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306089665 00000100:00000001:1.0:1787758361.795010:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306088971 00000100:00000001:1.0:1787758361.795013:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306089665 00000100:00000001:1.0:1787758361.795016:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758361.795018:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.795021:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758361.795026:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758361/real 1787758361] req@ffff8ffb45c31500 x1874588659740544/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758361.795036:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758361.795037:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758361.795040:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758361.795042:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758361.795044:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795046:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758361.795049:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758361.795056:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659740544/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758361.795071:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795075:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758361.795080:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795083:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758361.795086:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795089:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758361.795109:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758361.795115:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758361.795121:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758361.795126:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758361.795134:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740544:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758361.795142:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758361.795145:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758361.795148:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.795152:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758361.795156:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758361.795160:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758361.795166:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740672:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758361.795172:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758361.795176:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758361.795179:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758361.795243:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758361.795250:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.795258:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758361.795260:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787758361.795265:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758361.795284:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758361.795290:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758361.795294:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758361.795296:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.795298:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758361.795302:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.795306:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306088972 00000100:00000001:1.0:1787758361.795310:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306089665 00000100:00000001:1.0:1787758361.795312:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306088972 00000100:00000001:1.0:1787758361.795316:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306089665 00000100:00000001:1.0:1787758361.795318:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758361.795321:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.795323:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758361.795329:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758361/real 1787758361] req@ffff8ffb45c30000 x1874588659740672/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758361.795346:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758361.795348:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758361.795351:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758361.795353:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758361.795355:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795357:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758361.795359:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758361.795364:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659740672/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758361.795373:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795376:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758361.795378:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795380:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758361.795382:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795384:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758361.795397:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758361.795400:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758361.795404:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758361.795406:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758361.795410:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740672:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758361.795414:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795417:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758361.795419:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758361.795421:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758361.795424:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758361.795426:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758361.795428:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758361.795431:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758361.795433:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758361.795436:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758361.795438:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795441:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758361.795442:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758361.795444:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758361.795447:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758361.795449:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758361.795451:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758361.795452:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758361.795454:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758361.795456:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758361.795457:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795459:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758361.795460:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758361.795462:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758361.795465:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758361.795466:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758361.795468:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758361.795471:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758361.795473:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758361.795477:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758361.795479:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795483:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758361.795490:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758361.795493:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758361.795496:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758361.795500:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758362.369337:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758362.369344:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758362.369346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758362.369352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758362.369356:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758362.369360:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758362.369363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758362.369366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758362.369369:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758362.561327:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758362.561337:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758362.561346:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758362.561348:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758362.561407:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758362.561412:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758362.561416:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758363.393326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758363.393338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758363.393343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758363.393352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758363.393358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758363.393365:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758363.393371:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758363.393376:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758363.393381:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758363.585340:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758363.585349:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758363.585356:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758363.585358:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758363.585437:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758363.585443:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758363.585448:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758364.417305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758364.417314:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758364.417317:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758364.417325:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758364.417330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758364.417335:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758364.417338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758364.417341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758364.417345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758364.609336:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758364.609344:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758364.609353:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758364.609355:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758364.609523:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758364.609528:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758364.609534:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758365.441287:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758365.441295:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758365.441297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758365.441302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758365.441306:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758365.441310:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758365.441313:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758365.441315:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758365.441318:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758365.633308:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758365.633315:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758365.633322:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758365.633326:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758365.633328:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758365.633331:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758365.633334:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758365.633336:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758365.633337:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758365.633340:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758365.633361:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758365.633364:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758365.633367:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758366.465295:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758366.465307:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758366.465311:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758366.465319:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758366.465326:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758366.465332:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758366.465338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758366.465342:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758366.465347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758366.657308:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758366.657316:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758366.657324:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758366.657327:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758366.657367:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758366.657377:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758366.657384:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758366.913352:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758366.913362:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758366.913365:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758366.913368:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758366.913372:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758366.913379:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11421 00000100:00080000:2.0:1787758366.913390:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758366.913393:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758366.913395:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758366.913399:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758366.913401:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758366.913404:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758366.913407:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758366.913409:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758366.913410:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758366.913414:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758366.913417:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758366.913419:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758366.913423:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758366.913426:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913428:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758366.913430:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758366.913441:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758366.913443:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913446:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758366.913448:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758366.913452:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913463:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758366.913472:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38a80 x1874588659740800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758366.913483:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758366.913487:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913507:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758366.913510:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758366.913513:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758366.913515:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758366.913516:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758366.913518:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758366.913520:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11421 00000100:00080000:2.0:1787758366.913524:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758366.913525:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758366.913527:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758366.913528:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758366.913529:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758366.913531:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758366.913533:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758366.913535:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758366.913536:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758366.913539:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758366.913541:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758366.913542:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758366.913547:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758366.913548:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913550:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758366.913554:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758366.913558:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913560:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758366.913563:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758366.913565:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913567:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758366.913570:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758366.913572:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913576:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758366.913582:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39500 x1874588659740928/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758366.913589:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758366.913591:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913594:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.913596:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787758366.913598:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758366.913609:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758366.913614:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00080000:2.0:1787758366.913617:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758366.913620:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758366.913622:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758366.913625:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758366.913628:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758366.913635:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787758366.913638:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:2.0:1787758366.913639:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11421 02000000:00000001:1.0:1787758366.913647:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787758366.913648:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758366.913666:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758366.913667:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913677:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758366.913686:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00100000:1.0:1787758366.913689:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659740800:192.168.201.118@tcp:250:kworker.0 00000020:00000001:2.0:1787758366.913690:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758366.913697:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758366.913707:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758366.913720:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758366.913723:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1787758366.913724:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758366.913735:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787758366.913736:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758366.913742:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913746:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758366.913748:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758366.913756:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758366.913763:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758366.913770:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913773:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758366.913775:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758366.913777:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:2.0:1787758366.913779:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913788:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758366.913798:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758366.913802:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913804:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758366.913807:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758366.913811:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.913816:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758366.913819:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.913825:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134304384 : -123165575247232 : ffff8ffb47a38a80) 00000100:00100000:2.0:1787758366.913835:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659741056/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758366.913840:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758366.913860:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758366.913865:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913873:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758366.913877:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758366.913882:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758366.913884:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758366.913886:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758366.913891:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758366.913895:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11421 00000100:00080000:2.0:1787758366.913900:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758366.913902:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758366.913904:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758366.913906:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758366.913907:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758366.913909:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758366.913911:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758366.913914:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758366.913917:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:1.0:1787758366.913925:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1787758366.913933:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758366.913936:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758366.913938:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758366.913942:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758366.913944:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913949:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758366.913952:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758366.913956:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913958:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758366.913963:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758366.913965:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:2.0:1787758366.913966:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.913973:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758366.913975:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758366.913976:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:2.0:1787758366.913979:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.913982:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.913987:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758366.913988:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.913998:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758366.913999:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659741184/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758366.914010:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306094785 00000100:00000001:2.0:1787758366.914018:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758366.914022:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758366.914026:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914032:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758366.914043:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306094785 00000100:00000001:1.0:1787758366.914050:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758366.914057:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758366.914062:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914076:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758366.914089:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758366/real 1787758366] req@ffff8ffb47a38a80 x1874588659740800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758366.914107:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758366.914111:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758366.914117:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758366.914121:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758366.914153:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914157:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758366.914160:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758366.914179:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38a80 x1874588659740800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758366.914231:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914237:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758366.914241:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914244:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758366.914245:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914249:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758366.914252:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758366.914256:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758366.914262:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758366.914265:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758366.914271:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659740800:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758366.914278:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758366.914280:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758366.914282:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758366.914286:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758366.914290:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758366.914293:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758366.914297:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740928:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758366.914302:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758366.914305:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758366.914307:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758366.914309:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758366.914314:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914323:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758366.914325:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307072 : -123165575244544 : ffff8ffb47a39500) 00000100:00000001:1.0:1787758366.914332:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758366.914353:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758366.914360:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758366.914363:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758366.914365:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914368:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758366.914372:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914377:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306094094 00000100:00000001:1.0:1787758366.914381:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306094785 00000100:00000001:1.0:1787758366.914385:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306094094 00000100:00000001:1.0:1787758366.914388:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306094785 00000100:00000001:1.0:1787758366.914391:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758366.914394:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914396:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758366.914402:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758366/real 1787758366] req@ffff8ffb47a39500 x1874588659740928/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758366.914412:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758366.914414:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758366.914417:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758366.914419:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758366.914421:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914424:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758366.914426:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758366.914431:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39500 x1874588659740928/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758366.914439:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914442:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758366.914446:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914448:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758366.914450:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914452:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758366.914472:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758366.914475:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758366.914479:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758366.914482:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758366.914486:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659740928:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758366.914491:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914495:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758366.914498:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758366.914502:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758366.914508:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758366.914512:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758366.914515:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758366.914521:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758366.914523:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758366.914529:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758366.914530:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914533:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758366.914534:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758366.914536:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758366.914540:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758366.914541:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758366.914543:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758366.914545:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758366.914546:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758366.914550:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758366.914551:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914554:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914565:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758366.914567:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914570:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758366.914573:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758366.914574:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758366.914576:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758366.914578:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758366.914582:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758366.914583:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758366.914586:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758366.914590:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741056:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758366.914595:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758366.914597:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758366.914599:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758366.914601:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758366.914604:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914611:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758366.914612:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787758366.914618:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758366.914632:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758366.914637:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758366.914640:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758366.914642:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914644:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758366.914647:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914651:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306094091 00000100:00000001:1.0:1787758366.914655:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306094785 00000100:00000001:1.0:1787758366.914658:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306094091 00000100:00000001:1.0:1787758366.914661:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306094785 00000100:00000001:1.0:1787758366.914664:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758366.914666:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914668:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758366.914673:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758366/real 1787758366] req@ffff8ffb47a3a680 x1874588659741056/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758366.914682:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758366.914684:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758366.914686:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758366.914689:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758366.914691:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914693:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758366.914695:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758366.914700:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659741056/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758366.914708:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914711:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758366.914714:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914715:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758366.914717:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914719:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758366.914737:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758366.914740:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758366.914745:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758366.914748:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758366.914752:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741056:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758366.914757:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758366.914759:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758366.914761:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758366.914764:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758366.914766:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758366.914769:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758366.914775:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741184:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758366.914781:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758366.914784:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758366.914787:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758366.914790:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758366.914795:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914804:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758366.914807:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:1.0:1787758366.914815:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758366.914837:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758366.914850:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758366.914854:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758366.914857:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914861:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758366.914866:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914873:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306094091 00000100:00000001:1.0:1787758366.914878:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306094785 00000100:00000001:1.0:1787758366.914882:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306094091 00000100:00000001:1.0:1787758366.914887:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306094785 00000100:00000001:1.0:1787758366.914890:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758366.914892:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.914895:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758366.914903:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758366/real 1787758366] req@ffff8ffb47a39880 x1874588659741184/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758366.914913:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758366.914916:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758366.914919:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758366.914922:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758366.914923:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914926:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758366.914927:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758366.914932:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659741184/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758366.914940:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914943:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758366.914946:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914948:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758366.914949:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914951:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758366.914970:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758366.914973:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758366.914978:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758366.914981:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758366.914985:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741184:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758366.914989:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.914991:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758366.914993:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758366.914995:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758366.914998:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758366.915000:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758366.915002:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758366.915007:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758366.915008:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758366.915012:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758366.915015:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.915017:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758366.915019:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758366.915020:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758366.915023:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758366.915024:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758366.915026:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758366.915028:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758366.915029:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758366.915031:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758366.915032:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.915035:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758366.915039:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758366.915041:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758366.915043:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758366.915045:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758367.489265:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758367.489275:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758367.489278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758367.489284:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758367.489289:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758367.489294:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758367.489298:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758367.489301:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758367.489304:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758367.681318:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758367.681329:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758367.681334:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758367.681338:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758367.681345:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758367.681348:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758367.681352:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758367.681356:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758367.681362:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758367.681364:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758367.681431:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758367.681437:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758367.681441:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758368.513256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758368.513265:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758368.513268:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758368.513275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758368.513280:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758368.513284:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758368.513287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758368.513290:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758368.513293:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758368.705328:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758368.705338:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758368.705346:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758368.705349:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758368.705444:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758368.705451:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758368.705456:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758369.537268:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758369.537277:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758369.537280:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758369.537287:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758369.537291:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758369.537296:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758369.537299:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758369.537302:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758369.537306:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758369.729263:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758369.729271:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758369.729277:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758369.729278:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758369.729303:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758369.729307:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758369.729312:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758370.561297:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758370.561305:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758370.561307:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758370.561313:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758370.561316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758370.561320:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758370.561323:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758370.561326:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758370.561347:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758370.753301:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758370.753312:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758370.753323:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758370.753327:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758370.753390:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758370.753396:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758370.753401:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758371.585292:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758371.585301:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758371.585304:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758371.585311:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758371.585316:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758371.585321:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758371.585324:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758371.585328:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758371.585331:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758371.777398:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758371.777410:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758371.777419:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758371.777421:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758371.777535:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758371.777547:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758371.777555:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758372.033393:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758372.033411:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758372.033416:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758372.033422:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758372.033430:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758372.033441:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11426 00000100:00080000:3.0:1787758372.033460:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758372.033465:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758372.033469:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758372.033475:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758372.033479:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758372.033484:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758372.033505:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758372.033509:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758372.033511:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758372.033519:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758372.033524:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758372.033528:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758372.033535:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758372.033540:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033544:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758372.033548:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758372.033588:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758372.033592:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033600:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758372.033606:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758372.033615:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033640:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758372.033657:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659741312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758372.033673:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758372.033681:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033726:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758372.033730:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758372.033734:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758372.033736:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758372.033738:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758372.033741:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758372.033744:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11426 00000100:00080000:3.0:1787758372.033749:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758372.033751:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758372.033753:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758372.033755:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758372.033758:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758372.033762:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758372.033764:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758372.033766:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758372.033768:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758372.033773:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758372.033778:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758372.033781:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758372.033788:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758372.033791:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033794:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758372.033801:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758372.033804:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033806:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758372.033812:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758372.033813:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033815:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758372.033817:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758372.033819:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033824:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.033829:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00100000:3.0:1787758372.033832:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659741440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758372.033849:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758372.033850:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758372.033858:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.033862:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758372.033865:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758372.033874:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758372.033876:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00080000:3.0:1787758372.033884:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758372.033888:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.033888:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758372.033894:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758372.033898:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758372.033906:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11426 02000000:00000001:1.0:1787758372.033909:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787758372.033910:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758372.033918:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758372.033922:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758372.033924:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758372.033926:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758372.033930:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758372.033935:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758372.033940:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758372.033940:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758372.033955:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758372.033959:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758372.033971:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758372.033976:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758372.033979:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758372.033983:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758372.033985:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758372.033987:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659741312:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758372.033989:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758372.033999:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758372.034004:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:3.0:1787758372.034007:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.034011:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.034016:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1787758372.034017:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758372.034023:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.034026:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.034030:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758372.034057:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:3.0:1787758372.034058:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758372.034067:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.034078:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.034081:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758372.034099:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659741568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758372.034112:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758372.034116:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758372.034120:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:3.0:1787758372.034124:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034133:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758372.034136:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758372.034144:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758372.034150:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758372.034153:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758372.034157:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758372.034163:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758372.034169:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11426 00000100:00080000:3.0:1787758372.034233:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758372.034238:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758372.034240:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758372.034244:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758372.034246:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758372.034249:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758372.034252:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000800:00000100:1.0:1787758372.034277:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758372.034288:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758372.034293:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758372.034302:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758372.034307:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758372.034311:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758372.034317:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787758372.034318:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758372.034326:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.034329:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758372.034330:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000008:00100000:3.0:1787758372.034334:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758372.034336:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787758372.034356:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034358:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758372.034371:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034376:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758372.034384:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758372.034386:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306099906 02000000:00000001:3.0:1787758372.034389:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.034398:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758372.034401:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758372.034405:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034408:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758372.034416:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034420:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306099906 00000100:00000001:1.0:1787758372.034429:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00100000:3.0:1787758372.034430:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659741696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758372.034443:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758372.034460:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.034460:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758372.034476:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758372.034478:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.034495:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758372.034512:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758372/real 1787758372] req@ffff8ffb45c33800 x1874588659741312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758372.034536:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758372.034540:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758372.034548:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758372.034551:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758372.034553:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034557:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758372.034561:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758372.034570:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659741312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758372.034592:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034596:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758372.034601:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034603:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758372.034604:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034607:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758372.034610:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758372.034614:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758372.034620:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758372.034623:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758372.034629:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659741312:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758372.034635:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034638:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758372.034641:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758372.034644:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758372.034649:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758372.034653:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758372.034655:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758372.034660:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758372.034662:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758372.034666:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758372.034667:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034671:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034679:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758372.034682:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034697:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758372.034700:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758372.034702:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758372.034704:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758372.034706:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.034710:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758372.034713:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758372.034716:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758372.034720:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741440:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758372.034725:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758372.034728:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758372.034730:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.034733:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758372.034736:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034745:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758372.034748:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:1.0:1787758372.034756:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758372.034784:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758372.034793:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758372.034798:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758372.034801:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034805:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758372.034810:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034818:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306099214 00000100:00000001:1.0:1787758372.034824:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306099906 00000100:00000001:1.0:1787758372.034829:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306099214 00000100:00000001:1.0:1787758372.034833:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306099906 00000100:00000001:1.0:1787758372.034837:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758372.034842:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.034846:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758372.034855:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758372/real 1787758372] req@ffff8ffb45c33480 x1874588659741440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758372.034872:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758372.034875:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758372.034881:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758372.034885:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758372.034887:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034892:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758372.034895:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758372.034904:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659741440/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758372.034919:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034924:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758372.034930:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034945:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758372.034948:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.034952:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758372.034972:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758372.034976:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758372.034981:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758372.034984:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758372.034990:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741440:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758372.034995:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758372.034998:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758372.035000:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.035004:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758372.035006:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758372.035009:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758372.035016:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758372.035023:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758372.035027:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758372.035030:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.035058:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758372.035074:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035087:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758372.035090:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787758372.035097:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758372.035125:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758372.035135:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758372.035139:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758372.035141:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035144:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758372.035148:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035153:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306099211 00000100:00000001:1.0:1787758372.035157:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306099906 00000100:00000001:1.0:1787758372.035160:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306099211 00000100:00000001:1.0:1787758372.035163:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306099906 00000100:00000001:1.0:1787758372.035166:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758372.035168:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035171:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758372.035177:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758372/real 1787758372] req@ffff8ffb45c31180 x1874588659741568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758372.035230:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758372.035232:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758372.035235:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758372.035238:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758372.035239:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035242:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758372.035244:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758372.035249:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659741568/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758372.035258:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035261:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758372.035265:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035266:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758372.035268:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035270:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758372.035286:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758372.035289:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758372.035293:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758372.035295:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758372.035299:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741568:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758372.035309:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758372.035311:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758372.035314:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.035317:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758372.035320:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758372.035322:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758372.035326:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758372.035330:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758372.035333:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758372.035336:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758372.035338:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758372.035342:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035349:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758372.035351:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:1.0:1787758372.035356:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758372.035372:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758372.035378:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758372.035381:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758372.035383:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035385:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758372.035388:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035392:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306099212 00000100:00000001:1.0:1787758372.035396:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306099906 00000100:00000001:1.0:1787758372.035398:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306099212 00000100:00000001:1.0:1787758372.035401:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306099906 00000100:00000001:1.0:1787758372.035404:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758372.035407:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035409:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758372.035414:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758372/real 1787758372] req@ffff8ffb45c30000 x1874588659741696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758372.035423:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758372.035424:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758372.035426:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758372.035428:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758372.035430:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035432:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758372.035434:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758372.035439:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659741696/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758372.035447:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035450:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758372.035454:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035456:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758372.035457:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035459:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758372.035475:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758372.035478:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758372.035482:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758372.035485:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758372.035489:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741696:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758372.035493:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035496:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758372.035497:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758372.035500:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758372.035503:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758372.035507:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758372.035508:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758372.035511:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758372.035513:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758372.035516:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758372.035517:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035519:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758372.035521:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758372.035523:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758372.035526:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758372.035527:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758372.035528:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758372.035530:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758372.035531:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758372.035533:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758372.035535:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035537:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758372.035538:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758372.035539:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758372.035541:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758372.035542:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758372.035544:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758372.035546:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758372.035547:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758372.035550:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758372.035551:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035554:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758372.035559:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758372.035561:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758372.035564:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758372.035566:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758372.609301:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758372.609311:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758372.609314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758372.609320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758372.609325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758372.609330:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758372.609333:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758372.609337:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758372.609340:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758372.801318:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758372.801326:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758372.801334:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758372.801336:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758372.801396:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758372.801401:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758372.801405:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758373.633284:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758373.633294:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758373.633297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758373.633303:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758373.633307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758373.633312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758373.633315:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758373.633318:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758373.633321:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758373.825295:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758373.825304:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758373.825312:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758373.825315:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758373.825345:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758373.825350:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758373.825355:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758374.657244:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758374.657249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758374.657251:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758374.657255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758374.657258:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758374.657260:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758374.657262:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758374.657264:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758374.657265:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758374.849253:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758374.849258:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758374.849263:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758374.849264:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758374.849345:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758374.849349:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758374.849352:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758375.681269:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758375.681277:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758375.681280:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758375.681286:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758375.681290:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758375.681295:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758375.681297:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758375.681300:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758375.681303:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758375.873310:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758375.873318:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758375.873326:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758375.873330:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758375.873332:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758375.873336:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758375.873339:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758375.873341:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758375.873344:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758375.873346:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758375.873469:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758375.873474:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758375.873480:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758376.705255:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758376.705264:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758376.705267:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758376.705272:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758376.705276:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758376.705280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758376.705282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758376.705285:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758376.705287:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758376.897365:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758376.897373:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758376.897380:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758376.897382:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758376.897448:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758376.897458:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758376.897465:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758377.153271:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758377.153286:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758377.153289:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758377.153295:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758377.153301:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758377.153311:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11431 00000100:00080000:3.0:1787758377.153325:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758377.153329:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758377.153332:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758377.153338:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758377.153342:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758377.153347:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758377.153350:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758377.153353:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758377.153355:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758377.153361:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758377.153364:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758377.153367:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758377.153371:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758377.153375:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153378:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758377.153381:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758377.153394:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758377.153396:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153401:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758377.153404:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758377.153410:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153426:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758377.153439:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659741824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758377.153452:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758377.153458:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153477:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758377.153480:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758377.153485:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758377.153487:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758377.153489:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758377.153492:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758377.153495:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11431 00000100:00080000:3.0:1787758377.153498:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758377.153500:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758377.153503:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758377.153505:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758377.153507:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758377.153509:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758377.153512:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758377.153515:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758377.153518:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758377.153521:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758377.153524:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758377.153528:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758377.153535:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758377.153538:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153541:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758377.153548:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758377.153552:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.153556:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758377.153556:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758377.153562:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758377.153570:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.153571:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758377.153579:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758377.153581:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758377.153585:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758377.153586:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758377.153590:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758377.153591:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.153608:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758377.153609:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.153614:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758377.153620:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758377.153624:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659741952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758377.153634:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659741824:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758377.153659:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:3.0:1787758377.153666:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758377.153670:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758377.153672:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.153676:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.153683:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758377.153687:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.153694:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758377.153704:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758377.153716:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787758377.153719:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758377.153720:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:3.0:1787758377.153727:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758377.153731:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00080000:3.0:1787758377.153734:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758377.153742:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758377.153749:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11431 00000100:00080000:3.0:1787758377.153756:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758377.153758:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758377.153760:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758377.153765:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758377.153767:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758377.153771:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758377.153774:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758377.153778:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758377.153780:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000800:00000100:1.0:1787758377.153789:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1787758377.153796:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758377.153802:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758377.153806:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758377.153810:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787758377.153811:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758377.153817:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.153818:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758377.153825:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153827:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758377.153835:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000008:00100000:3.0:1787758377.153838:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758377.153844:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787758377.153846:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153859:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.153860:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306105025 02000000:00000001:3.0:1787758377.153882:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758377.153886:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787758377.153888:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153894:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758377.153896:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306105025 02000000:00000001:3.0:1787758377.153897:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758377.153908:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.153910:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758377.153918:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758377.153923:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153923:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758377.153964:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659742080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758377.153971:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758377.153982:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00100000:1.0:1787758377.153987:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758377/real 1787758377] req@ffff8ffb45c31500 x1874588659741824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758377.153988:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.153999:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758377.154013:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000400:00000001:1.0:1787758377.154034:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00080000:3.0:1787758377.154034:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000400:00000001:1.0:1787758377.154040:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758377.154042:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758377.154047:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758377.154048:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758377.154058:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758377.154060:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758377.154062:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154068:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:3.0:1787758377.154070:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11431 00000100:00000001:1.0:1787758377.154073:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758377.154084:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758377.154088:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659741824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758377.154092:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758377.154098:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758377.154109:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758377.154115:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758377.154121:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1787758377.154124:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758377.154128:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787758377.154130:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758377.154138:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758377.154140:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758377.154143:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758377.154146:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758377.154147:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154159:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758377.154167:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:3.0:1787758377.154169:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758377.154226:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758377.154230:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758377.154236:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758377.154238:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:3.0:1787758377.154240:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.154248:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00080000:1.0:1787758377.154255:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1787758377.154256:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758377.154269:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154270:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787758377.154273:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758377.154284:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758377.154289:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758377.154290:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659741824:192.168.201.118@tcp:250:kworker.0 00000100:00000001:3.0:1787758377.154293:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758377.154298:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758377.154303:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154310:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758377.154313:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154320:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758377.154328:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00100000:3.0:1787758377.154330:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659742208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758377.154335:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758377.154347:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787758377.154348:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000020:00000001:1.0:1787758377.154357:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:3.0:1787758377.154358:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758377.154365:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787758377.154368:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.154371:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758377.154375:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758377.154380:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758377.154383:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154387:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154397:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758377.154400:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154403:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758377.154406:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758377.154408:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758377.154411:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758377.154413:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.154416:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758377.154419:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758377.154423:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758377.154429:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741952:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758377.154434:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758377.154438:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758377.154440:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.154442:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758377.154446:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154455:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758377.154457:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787758377.154463:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758377.154483:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758377.154490:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758377.154493:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758377.154495:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154497:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758377.154501:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154506:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306104334 00000100:00000001:1.0:1787758377.154510:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306105025 00000100:00000001:1.0:1787758377.154513:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306104334 00000100:00000001:1.0:1787758377.154516:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306105025 00000100:00000001:1.0:1787758377.154518:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758377.154521:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154524:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758377.154530:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758377/real 1787758377] req@ffff8ffb45c32a00 x1874588659741952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758377.154542:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758377.154543:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758377.154546:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758377.154549:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758377.154550:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154552:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758377.154554:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758377.154559:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659741952/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758377.154568:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154571:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758377.154574:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154575:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758377.154577:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154579:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758377.154594:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758377.154597:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758377.154601:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758377.154604:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758377.154608:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659741952:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758377.154613:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758377.154615:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758377.154617:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.154620:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758377.154622:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758377.154624:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758377.154628:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758377.154635:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758377.154637:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758377.154639:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.154641:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758377.154645:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154650:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758377.154652:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787758377.154657:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758377.154669:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758377.154673:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758377.154676:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758377.154678:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154680:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758377.154683:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154687:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306104331 00000100:00000001:1.0:1787758377.154691:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306105025 00000100:00000001:1.0:1787758377.154693:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306104331 00000100:00000001:1.0:1787758377.154696:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306105025 00000100:00000001:1.0:1787758377.154699:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758377.154701:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154703:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758377.154708:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758377/real 1787758377] req@ffff8ffb45c30700 x1874588659742080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758377.154717:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758377.154718:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758377.154721:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758377.154722:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758377.154724:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154726:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758377.154727:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758377.154732:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659742080/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758377.154748:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154751:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758377.154754:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154756:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758377.154757:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154759:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758377.154771:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758377.154774:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758377.154777:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758377.154780:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758377.154783:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742080:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758377.154788:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758377.154790:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758377.154792:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.154794:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758377.154797:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758377.154799:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758377.154802:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758377.154806:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758377.154808:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758377.154809:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758377.154812:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758377.154815:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154819:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758377.154821:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:1.0:1787758377.154825:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758377.154835:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758377.154840:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758377.154842:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758377.154844:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154847:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758377.154849:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154853:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306104331 00000100:00000001:1.0:1787758377.154856:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306105025 00000100:00000001:1.0:1787758377.154859:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306104331 00000100:00000001:1.0:1787758377.154862:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306105025 00000100:00000001:1.0:1787758377.154864:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758377.154867:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.154869:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758377.154874:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758377/real 1787758377] req@ffff8ffb45c30a80 x1874588659742208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758377.154882:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758377.154884:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758377.154886:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758377.154888:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758377.154889:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154891:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758377.154893:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758377.154897:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659742208/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758377.154906:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154908:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758377.154911:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154913:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758377.154915:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154917:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758377.154929:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758377.154932:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758377.154935:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758377.154938:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758377.154942:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742208:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758377.154971:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154974:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758377.154975:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758377.154977:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758377.154980:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758377.154982:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758377.154984:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758377.154987:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758377.154989:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758377.154991:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758377.154993:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.154995:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758377.154997:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758377.154998:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758377.155002:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758377.155003:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758377.155004:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758377.155006:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758377.155007:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758377.155009:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758377.155011:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.155013:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758377.155014:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758377.155015:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758377.155018:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758377.155019:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758377.155020:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758377.155022:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758377.155023:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758377.155025:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758377.155026:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.155029:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758377.155033:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758377.155034:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758377.155037:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758377.155039:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758377.729315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758377.729327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758377.729330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758377.729339:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758377.729358:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758377.729364:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758377.729368:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758377.729373:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758377.729377:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758377.921373:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758377.921388:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758377.921394:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758377.921404:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758377.921413:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758377.921419:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758377.921424:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758377.921429:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758377.921439:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758377.921442:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758377.921484:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758377.921488:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758377.921491:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758378.753306:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758378.753315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758378.753318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758378.753324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758378.753329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758378.753334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758378.753338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758378.753341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758378.753345:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758378.945370:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758378.945380:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758378.945388:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758378.945390:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758378.945506:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758378.945512:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758378.945517:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758379.777246:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758379.777269:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758379.777271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758379.777275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758379.777278:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758379.777282:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758379.777284:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758379.777286:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758379.777288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758379.969318:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758379.969327:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758379.969336:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758379.969338:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758379.969378:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758379.969381:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758379.969383:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758380.801260:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758380.801267:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758380.801269:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758380.801273:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758380.801276:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758380.801279:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758380.801281:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758380.801284:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758380.801286:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758380.993317:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758380.993325:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758380.993332:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758380.993334:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758380.993476:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758380.993482:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758380.993487:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758381.825369:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758381.825382:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758381.825384:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758381.825393:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758381.825399:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758381.825404:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758381.825408:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758381.825412:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758381.825416:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758382.017332:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758382.017341:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758382.017349:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758382.017351:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758382.017381:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758382.017385:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758382.017389:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758382.273387:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758382.273398:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758382.273402:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758382.273407:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758382.273415:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758382.273425:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11437 00000100:00080000:3.0:1787758382.273440:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758382.273445:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758382.273448:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758382.273454:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758382.273459:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758382.273464:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758382.273468:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758382.273471:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758382.273473:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758382.273480:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758382.273484:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758382.273488:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758382.273505:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758382.273509:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.273512:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758382.273515:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758382.273528:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758382.273530:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.273534:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758382.273538:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758382.273544:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.273558:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758382.273571:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659742336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758382.273585:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758382.273590:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.273617:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758382.273622:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758382.273628:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758382.273631:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758382.273634:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758382.273640:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758382.273645:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11437 00000100:00080000:3.0:1787758382.273650:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758382.273653:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758382.273656:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758382.273662:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758382.273664:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758382.273668:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758382.273671:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758382.273675:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758382.273678:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758382.273683:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758382.273686:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758382.273689:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758382.273697:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758382.273700:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.273704:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758382.273710:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758382.273715:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.273718:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758382.273719:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.273728:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:3.0:1787758382.273731:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758382.273734:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758382.273737:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.273742:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758382.273744:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758382.273750:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758382.273751:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758382.273757:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758382.273762:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758382.273768:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758382.273770:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758382.273778:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758382.273786:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30e00 x1874588659742464/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758382.273793:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659742336:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758382.273812:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758382.273821:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758382.273826:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758382.273830:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758382.273837:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.273887:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758382.273890:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787758382.273896:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758382.273950:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758382.273967:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758382.273973:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758382.273977:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.273980:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758382.273987:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758382.273989:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.273996:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274000:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306110145 00000100:00000001:3.0:1787758382.274006:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758382.274013:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758382.274015:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758382.274025:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306110145 00000100:00000001:1.0:1787758382.274030:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758382.274034:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758382.274038:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274041:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758382.274049:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758382/real 1787758382] req@ffff8ffb45c33b80 x1874588659742336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758382.274065:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758382.274068:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758382.274072:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758382.274076:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758382.274078:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274082:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758382.274085:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758382.274090:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659742336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758382.274101:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274106:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758382.274110:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274113:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758382.274114:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274117:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758382.274120:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758382.274124:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758382.274130:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758382.274135:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758382.274139:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659742336:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758382.274145:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274148:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758382.274152:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758382.274157:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758382.274164:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758382.274169:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758382.274174:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00080000:3.0:1787758382.274211:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758382.274216:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758382.274219:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758382.274224:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787758382.274256:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758382.274263:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758382.274271:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758382.274275:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274281:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758382.274292:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11437 00000100:00000001:1.0:1787758382.274293:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758382.274300:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758382.274306:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758382.274309:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758382.274316:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758382.274318:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758382.274321:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758382.274325:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000020:00000001:3.0:1787758382.274328:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758382.274332:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000020:00000001:3.0:1787758382.274333:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758382.274339:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758382.274341:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758382.274352:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758382.274353:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758382.274362:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758382.274363:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758382.274366:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758382.274370:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758382.274378:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1787758382.274383:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742464:192.168.201.118@tcp:38:kworker.0 00000100:00000001:3.0:1787758382.274385:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758382.274394:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758382.274398:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:3.0:1787758382.274403:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758382.274406:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758382.274413:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758382.274414:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758382.274421:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758382.274422:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758382.274429:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758382.274431:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000008:00000001:3.0:1787758382.274434:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.274440:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274446:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787758382.274448:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758382.274452:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102815232 : -123165606736384 : ffff8ffb45c30e00) 02000000:00000001:3.0:1787758382.274454:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.274465:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758382.274468:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:3.0:1787758382.274469:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758382.274476:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.274488:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758382.274497:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31c00 x1874588659742592/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758382.274508:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758382.274520:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758382.274523:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758382.274525:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274533:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758382.274536:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274539:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758382.274543:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758382.274546:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787758382.274553:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758382.274557:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.274559:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758382.274566:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758382.274569:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306109454 00000100:00000001:3.0:1787758382.274570:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758382.274579:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11437 00000100:00000001:1.0:1787758382.274580:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306110145 00000100:00080000:3.0:1787758382.274588:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758382.274589:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306109454 00000100:00000001:3.0:1787758382.274598:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758382.274600:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306110145 00000100:00000001:3.0:1787758382.274605:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758382.274607:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1787758382.274613:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758382.274616:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758382.274619:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758382.274622:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000020:00000001:3.0:1787758382.274624:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758382.274631:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:1.0:1787758382.274636:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758382/real 1787758382] req@ffff8ffb45c30e00 x1874588659742464/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758382.274637:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758382.274643:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758382.274657:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758382.274667:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758382.274675:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000400:00000001:1.0:1787758382.274684:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787758382.274685:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000400:00000001:1.0:1787758382.274692:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 02000000:00000001:3.0:1787758382.274694:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.274700:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758382.274701:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000008:00100000:3.0:1787758382.274704:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758382.274708:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000008:00000001:3.0:1787758382.274709:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274716:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758382.274717:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274727:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:3.0:1787758382.274731:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758382.274733:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758382.274738:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.274744:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:1.0:1787758382.274747:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30e00 x1874588659742464/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758382.274749:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758382.274755:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274766:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274769:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758382.274772:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274774:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758382.274777:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274783:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758382.274798:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274800:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758382.274811:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758382.274819:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1787758382.274821:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33b80 x1874588659742720/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758382.274823:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758382.274838:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742464:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758382.274883:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758382.274885:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758382.274892:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758382.274895:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274897:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758382.274902:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758382.274910:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787758382.274910:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758382.274916:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758382.274920:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758382.274925:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758382.274928:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758382.274931:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758382.274934:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274936:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.274941:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758382.274943:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274946:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758382.274951:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758382.274954:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758382.274957:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758382.274960:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758382.274963:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758382.274966:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758382.274970:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758382.274976:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758382.274980:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758382.274983:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758382.274985:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758382.274988:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758382.274992:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.274999:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758382.275001:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102818816 : -123165606732800 : ffff8ffb45c31c00) 00000100:00000001:1.0:1787758382.275006:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758382.275023:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758382.275029:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758382.275032:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758382.275033:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.275036:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758382.275039:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.275043:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306109451 00000100:00000001:1.0:1787758382.275047:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306110145 00000100:00000001:1.0:1787758382.275050:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306109451 00000100:00000001:1.0:1787758382.275053:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306110145 00000100:00000001:1.0:1787758382.275055:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758382.275057:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.275060:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758382.275065:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758382/real 1787758382] req@ffff8ffb45c31c00 x1874588659742592/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758382.275075:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758382.275076:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758382.275079:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758382.275081:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758382.275083:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275085:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758382.275087:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758382.275092:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31c00 x1874588659742592/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758382.275101:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275104:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758382.275107:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275109:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758382.275111:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275113:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758382.275126:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758382.275129:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758382.275132:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758382.275135:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758382.275138:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742592:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758382.275143:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758382.275145:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758382.275147:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758382.275150:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758382.275152:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758382.275155:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758382.275161:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758382.275166:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758382.275170:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758382.275172:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758382.275176:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758382.275234:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.275249:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758382.275252:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102826880 : -123165606724736 : ffff8ffb45c33b80) 00000100:00000001:1.0:1787758382.275260:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758382.275286:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758382.275293:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758382.275296:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758382.275298:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.275301:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758382.275305:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.275309:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306109452 00000100:00000001:1.0:1787758382.275313:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306110145 00000100:00000001:1.0:1787758382.275315:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306109452 00000100:00000001:1.0:1787758382.275318:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306110145 00000100:00000001:1.0:1787758382.275321:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758382.275323:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.275326:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758382.275331:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758382/real 1787758382] req@ffff8ffb45c33b80 x1874588659742720/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758382.275340:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758382.275342:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758382.275345:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758382.275347:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758382.275348:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275351:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758382.275352:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758382.275357:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33b80 x1874588659742720/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758382.275366:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275369:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758382.275372:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275374:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758382.275375:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275377:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758382.275391:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758382.275394:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758382.275397:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758382.275399:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758382.275404:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742720:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758382.275408:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275411:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758382.275412:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758382.275415:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758382.275418:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758382.275420:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758382.275421:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758382.275424:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758382.275425:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758382.275428:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758382.275429:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275431:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758382.275433:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758382.275435:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758382.275437:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758382.275439:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758382.275440:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758382.275442:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758382.275443:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758382.275445:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758382.275446:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275449:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758382.275453:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758382.275455:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758382.275457:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758382.275459:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758382.849302:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758382.849311:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758382.849314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758382.849320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758382.849325:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758382.849329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758382.849333:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758382.849336:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758382.849340:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758383.041306:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758383.041314:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758383.041322:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758383.041324:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758383.041363:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758383.041370:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758383.041375:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758383.873267:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758383.873273:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758383.873275:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758383.873279:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758383.873281:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758383.873284:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758383.873286:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758383.873288:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758383.873289:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758384.065291:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758384.065298:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758384.065305:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758384.065306:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758384.065327:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758384.065330:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758384.065332:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758384.897289:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758384.897298:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758384.897301:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758384.897307:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758384.897312:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758384.897317:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758384.897321:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758384.897324:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758384.897328:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758385.089327:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758385.089335:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758385.089343:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758385.089345:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758385.089420:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758385.089425:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758385.089430:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758385.921334:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758385.921343:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758385.921346:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758385.921352:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758385.921357:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758385.921362:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758385.921366:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758385.921369:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758385.921373:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758386.113412:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758386.113420:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758386.113427:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758386.113431:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758386.113434:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758386.113438:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758386.113441:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758386.113444:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758386.113446:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758386.113449:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758386.113475:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758386.113478:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758386.113482:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758386.945256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758386.945263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758386.945264:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758386.945269:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758386.945272:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758386.945275:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758386.945277:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758386.945279:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758386.945281:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758387.137314:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758387.137322:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758387.137325:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758387.137329:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758387.137334:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758387.137337:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758387.137339:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758387.137342:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758387.137347:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758387.137349:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758387.137413:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758387.137418:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758387.137422:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758387.393307:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758387.393328:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758387.393331:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758387.393338:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758387.393343:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758387.393349:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11442 00000100:00080000:3.0:1787758387.393360:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758387.393363:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758387.393365:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758387.393369:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758387.393371:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758387.393374:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758387.393376:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758387.393378:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758387.393379:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758387.393384:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758387.393386:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758387.393388:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758387.393391:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758387.393394:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393396:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758387.393398:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758387.393408:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758387.393410:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393413:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758387.393415:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758387.393420:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393430:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758387.393439:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659742848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758387.393448:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758387.393452:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393468:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758387.393471:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758387.393474:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758387.393476:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758387.393477:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758387.393480:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758387.393481:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11442 00000100:00080000:3.0:1787758387.393484:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758387.393486:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758387.393487:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758387.393493:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758387.393495:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758387.393497:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758387.393499:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758387.393501:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758387.393502:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758387.393505:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758387.393507:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758387.393508:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758387.393513:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758387.393515:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393525:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758387.393530:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758387.393531:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00000001:3.0:1787758387.393533:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393537:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758387.393538:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393543:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758387.393545:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758387.393546:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758387.393548:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393550:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393552:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758387.393556:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758387.393557:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758387.393559:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758387.393562:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758387.393565:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758387.393566:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758387.393579:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659742848:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787758387.393580:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659742976/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758387.393588:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758387.393590:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393596:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393597:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:3.0:1787758387.393601:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787758387.393604:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:3.0:1787758387.393605:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:1.0:1787758387.393609:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393610:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758387.393614:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787758387.393615:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758387.393619:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758387.393623:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11442 02000000:00000001:1.0:1787758387.393624:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758387.393627:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758387.393630:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758387.393632:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758387.393634:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758387.393635:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758387.393637:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758387.393639:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758387.393641:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758387.393641:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758387.393643:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758387.393645:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00080000:3.0:1787758387.393650:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758387.393653:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758387.393655:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758387.393658:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758387.393661:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758387.393663:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393665:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758387.393667:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758387.393669:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393671:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758387.393675:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758387.393676:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393678:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758387.393679:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758387.393681:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393686:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758387.393690:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659743104/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758387.393695:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758387.393703:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758387.393705:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393710:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758387.393712:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758387.393716:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758387.393717:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758387.393718:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758387.393722:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758387.393725:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758387.393725:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758387.393731:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758387.393732:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11442 00000100:00000001:1.0:1787758387.393734:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00080000:3.0:1787758387.393739:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758387.393742:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758387.393743:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393746:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758387.393749:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306115265 00000020:00000001:3.0:1787758387.393750:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758387.393754:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758387.393757:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758387.393759:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758387.393761:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758387.393779:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758387.393783:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758387.393787:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393788:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 02000000:00000001:3.0:1787758387.393792:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758387.393796:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306115265 02000000:00000001:3.0:1787758387.393798:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758387.393802:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 02000000:00000001:3.0:1787758387.393805:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393818:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758387.393821:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758387.393822:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393827:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000008:00100000:3.0:1787758387.393827:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758387.393835:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758387/real 1787758387] req@ffff8ffb45c30a80 x1874588659742848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:3.0:1787758387.393836:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.393843:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758387.393854:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:1.0:1787758387.393855:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787758387.393859:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787758387.393860:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758387.393864:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758387.393866:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758387.393868:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:3.0:1787758387.393871:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758387.393872:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758387.393875:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393876:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758387.393881:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758387.393885:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758387.393887:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659742848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758387.393896:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659743232/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758387.393900:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.393906:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:3.0:1787758387.393911:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758387.393912:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758387.393916:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393917:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758387.393921:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758387.393922:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393925:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758387.393928:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758387.393932:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758387.393936:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758387.393938:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758387.393942:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659742848:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758387.393948:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.393950:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758387.393951:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758387.393953:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758387.393957:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758387.393959:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758387.393960:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758387.393963:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758387.393965:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758387.393967:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758387.393968:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.393970:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.393976:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758387.393978:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.393980:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758387.393981:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758387.393983:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758387.393984:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758387.393986:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758387.393988:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758387.393989:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758387.393991:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758387.393994:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742976:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758387.393997:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758387.393998:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758387.393999:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758387.394001:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758387.394004:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394008:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758387.394009:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:1.0:1787758387.394014:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758387.394024:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758387.394028:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758387.394030:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758387.394031:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394044:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758387.394047:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394050:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306114573 00000100:00000001:1.0:1787758387.394053:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306115265 00000100:00000001:1.0:1787758387.394055:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306114573 00000100:00000001:1.0:1787758387.394057:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306115265 00000100:00000001:1.0:1787758387.394059:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758387.394060:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394062:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758387.394066:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758387/real 1787758387] req@ffff8ffb45c30700 x1874588659742976/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758387.394072:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758387.394074:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758387.394075:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758387.394077:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758387.394078:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394080:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758387.394081:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758387.394084:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659742976/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758387.394090:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394092:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758387.394094:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394095:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758387.394097:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394098:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758387.394109:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758387.394111:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758387.394125:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758387.394127:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758387.394130:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659742976:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758387.394133:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758387.394134:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758387.394135:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758387.394137:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758387.394138:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758387.394140:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758387.394142:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758387.394145:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758387.394146:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758387.394148:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758387.394149:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758387.394152:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394156:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758387.394157:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:1.0:1787758387.394168:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758387.394178:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758387.394212:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758387.394214:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758387.394216:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394217:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758387.394219:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394222:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306114571 00000100:00000001:1.0:1787758387.394225:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306115265 00000100:00000001:1.0:1787758387.394227:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306114571 00000100:00000001:1.0:1787758387.394229:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306115265 00000100:00000001:1.0:1787758387.394231:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758387.394233:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394234:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758387.394238:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758387/real 1787758387] req@ffff8ffb45c32a00 x1874588659743104/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758387.394246:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758387.394247:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758387.394249:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758387.394251:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758387.394252:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394253:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758387.394255:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758387.394258:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659743104/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758387.394264:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394266:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758387.394268:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394269:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758387.394270:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394271:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758387.394290:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758387.394292:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758387.394294:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758387.394296:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758387.394298:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743104:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758387.394301:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758387.394302:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758387.394304:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758387.394306:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758387.394307:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758387.394309:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758387.394312:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758387.394314:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758387.394316:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758387.394317:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758387.394319:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758387.394322:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394325:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758387.394326:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00000001:1.0:1787758387.394330:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758387.394339:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758387.394342:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758387.394344:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758387.394345:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394347:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758387.394349:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394351:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306114571 00000100:00000001:1.0:1787758387.394353:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306115265 00000100:00000001:1.0:1787758387.394355:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306114571 00000100:00000001:1.0:1787758387.394357:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306115265 00000100:00000001:1.0:1787758387.394359:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758387.394361:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394362:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758387.394365:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758387/real 1787758387] req@ffff8ffb45c31500 x1874588659743232/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758387.394371:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758387.394372:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758387.394374:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758387.394376:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758387.394377:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394378:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758387.394380:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758387.394383:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659743232/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758387.394394:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394396:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758387.394398:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394400:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758387.394401:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394402:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758387.394412:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758387.394414:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758387.394416:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758387.394418:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758387.394420:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743232:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758387.394423:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394425:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758387.394426:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758387.394427:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758387.394429:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758387.394430:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758387.394431:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758387.394433:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758387.394434:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758387.394436:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758387.394437:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394438:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758387.394439:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758387.394441:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758387.394443:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758387.394444:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758387.394445:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758387.394446:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758387.394447:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758387.394448:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758387.394449:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394450:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758387.394451:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758387.394453:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758387.394454:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758387.394455:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758387.394456:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758387.394458:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758387.394459:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758387.394460:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758387.394461:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394463:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758387.394465:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758387.394466:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758387.394468:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758387.394469:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758387.969313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758387.969320:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758387.969322:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758387.969327:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758387.969330:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758387.969344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758387.969362:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758387.969364:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758387.969366:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758388.161259:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758388.161263:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758388.161268:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758388.161270:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758388.161286:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758388.161288:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758388.161290:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758388.993230:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758388.993236:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758388.993238:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758388.993242:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758388.993245:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758388.993248:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758388.993250:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758388.993252:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758388.993254:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758389.185253:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758389.185259:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758389.185263:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758389.185264:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758389.185352:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758389.185361:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758389.185367:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758390.017320:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758390.017327:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758390.017330:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758390.017335:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758390.017340:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758390.017344:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758390.017346:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758390.017350:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758390.017352:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758390.209323:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758390.209333:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758390.209341:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758390.209344:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758390.209392:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758390.209396:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758390.209400:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758391.041256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758391.041263:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758391.041264:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758391.041268:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758391.041271:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758391.041274:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758391.041276:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758391.041278:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758391.041280:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758391.233257:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758391.233263:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758391.233267:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758391.233268:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758391.233313:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758391.233316:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758391.233319:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758392.065296:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758392.065308:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758392.065312:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758392.065342:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758392.065346:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758392.065350:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758392.065353:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758392.065356:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758392.065359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758392.257314:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758392.257322:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758392.257328:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758392.257331:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758392.257353:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758392.257356:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758392.257358:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758392.513279:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758392.513287:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787758392.513289:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758392.513292:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758392.513296:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758392.513302:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11447 00000100:00080000:0.0:1787758392.513311:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758392.513314:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758392.513315:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758392.513318:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758392.513320:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758392.513322:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787758392.513324:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787758392.513326:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758392.513327:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758392.513330:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758392.513332:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758392.513334:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758392.513336:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758392.513339:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513340:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758392.513342:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758392.513352:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758392.513353:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513355:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758392.513357:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758392.513361:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513370:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758392.513377:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ca80 x1874588659743360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758392.513385:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758392.513388:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513400:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758392.513403:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758392.513405:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787758392.513406:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758392.513407:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758392.513409:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758392.513411:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11447 00000100:00080000:0.0:1787758392.513413:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758392.513415:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758392.513416:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758392.513418:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758392.513419:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758392.513421:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787758392.513423:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758392.513424:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758392.513425:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758392.513428:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758392.513428:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1787758392.513432:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513433:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:0.0:1787758392.513434:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758392.513438:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:0.0:1787758392.513441:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758392.513442:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758392.513445:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758392.513446:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513451:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 02000000:00000001:1.0:1787758392.513453:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000008:00100000:0.0:1787758392.513455:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758392.513457:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:0.0:1787758392.513458:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758392.513463:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513464:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758392.513470:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758392.513472:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758392.513474:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659743360:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758392.513475:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758392.513478:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758392.513481:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513482:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:0.0:1787758392.513486:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758392.513487:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758392.513491:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758392.513495:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:0.0:1787758392.513496:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790e680 x1874588659743488/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758392.513500:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513502:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758392.513505:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513507:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758392.513509:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758392.513511:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758392.513512:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758392.513513:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758392.513515:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758392.513516:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:0.0:1787758392.513517:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11447 00000100:00000001:1.0:1787758392.513520:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133075584 : -123165576476032 : ffff8ffb4790ca80) 00000100:00080000:0.0:1787758392.513522:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758392.513526:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758392.513526:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758392.513527:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758392.513531:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758392.513532:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758392.513534:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:0.0:1787758392.513535:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:0.0:1787758392.513537:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758392.513538:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758392.513540:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758392.513542:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758392.513543:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758392.513545:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758392.513546:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513547:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758392.513549:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758392.513552:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513553:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758392.513557:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758392.513557:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513559:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758392.513560:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758392.513561:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513565:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758392.513570:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790d180 x1874588659743616/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758392.513570:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513577:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306120385 00000100:00000001:0.0:1787758392.513585:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758392.513589:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513589:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787758392.513595:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513597:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306120385 00000100:00080000:0.0:1787758392.513601:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758392.513602:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758392.513607:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:0.0:1787758392.513608:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758392.513619:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513621:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758392.513624:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758392.513625:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758392.513627:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758392.513633:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758392.513637:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:0.0:1787758392.513638:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11447 00000100:00000001:1.0:1787758392.513640:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787758392.513645:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:0.0:1787758392.513647:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758392.513647:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758392.513650:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1787758392.513654:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758392.513654:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758392.513658:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513659:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758392.513661:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:0.0:1787758392.513665:0:87148:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:1.0:1787758392.513665:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000020:00000001:0.0:1787758392.513669:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 02000000:00000001:1.0:1787758392.513669:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758392.513686:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00100000:1.0:1787758392.513689:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743488:192.168.201.118@tcp:38:kworker.0 00000020:00000001:0.0:1787758392.513691:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758392.513696:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00000001:0.0:1787758392.513697:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758392.513699:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:0.0:1787758392.513701:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758392.513702:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758392.513706:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:0.0:1787758392.513709:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:1.0:1787758392.513710:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513717:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758392.513719:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133082752 : -123165576468864 : ffff8ffb4790e680) 00000100:00000001:0.0:1787758392.513731:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513733:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:0.0:1787758392.513736:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758392.513741:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758392.513744:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513745:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513746:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758392.513750:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306119693 00000008:00100000:0.0:1787758392.513751:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758392.513755:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306120385 00000008:00000001:0.0:1787758392.513757:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513759:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306119693 00000100:00000001:0.0:1787758392.513761:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513763:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306120385 00000100:00000001:1.0:1787758392.513766:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758392.513767:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758392.513768:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758392.513772:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513773:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758392.513775:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1787758392.513777:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758392.513778:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758392.513781:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758392.513782:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1787758392.513787:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758392.513787:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758392.513788:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758392.513793:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:0.0:1787758392.513795:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513796:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758392.513799:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758392.513801:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758392.513802:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00100000:0.0:1787758392.513806:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790fb80 x1874588659743744/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758392.513807:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513813:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758392.513816:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133077376 : -123165576474240 : ffff8ffb4790d180) 00000100:00000001:1.0:1787758392.513822:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758392.513824:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758392.513828:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513829:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758392.513832:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513833:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306119690 00000100:00000001:1.0:1787758392.513838:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306120385 00000100:00000001:1.0:1787758392.513840:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306119690 00000100:00000001:1.0:1787758392.513843:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306120385 00000100:00000001:1.0:1787758392.513844:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758392.513846:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513847:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513849:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758392.513851:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758392.513853:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787758392.513854:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758392.513856:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758392.513857:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758392.513858:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758392.513859:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758392.513860:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758392.513861:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758392.513862:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758392.513865:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758392.513867:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758392.513868:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758392.513869:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758392.513870:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758392.513872:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513874:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758392.513875:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133088128 : -123165576463488 : ffff8ffb4790fb80) 00000100:00000001:1.0:1787758392.513877:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:1.0:1787758392.513883:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513884:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306119690 00000100:00000001:1.0:1787758392.513886:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306120385 00000100:00000001:1.0:1787758392.513887:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306119690 00000100:00000001:1.0:1787758392.513889:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306120385 00000100:00000001:1.0:1787758392.513890:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758392.513891:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513893:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513894:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758392.513895:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758392.513896:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=5 : 5 : 5) 00000100:00000001:1.0:1787758392.513897:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758392.513897:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758392.513898:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758392.513900:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758393.089233:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758393.089239:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758393.089241:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758393.089246:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758393.089249:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758393.089252:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758393.089254:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758393.089256:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758393.089258:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758393.281282:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758393.281291:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758393.281298:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758393.281299:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758393.281342:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758393.281345:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758393.281348:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758394.113260:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758394.113268:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758394.113271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758394.113276:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758394.113280:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758394.113284:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758394.113287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758394.113290:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758394.113293:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758394.305295:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758394.305301:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758394.305306:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758394.305307:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758394.305324:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758394.305326:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758394.305329:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758395.137322:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758395.137331:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758395.137333:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758395.137340:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758395.137344:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758395.137347:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758395.137349:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758395.137352:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758395.137354:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758395.329310:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758395.329317:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758395.329322:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758395.329326:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758395.329329:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758395.329332:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758395.329347:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758395.329349:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758395.329351:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758395.329353:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758395.329409:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758395.329413:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758395.329416:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787758395.585543:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1787758395.585593:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787758395.585615:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787758395.585670:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758395.585693:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758395.585696:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758395.585699:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787758395.585711:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787758395.585718:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758395.585722:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758395.585723:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758395.585726:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000100:3.0:1787758395.585733:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:3.0:1787758395.585738:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:3.0:1787758395.585740:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758395.585742:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758395.585744:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758395.585749:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000800:00000100:3.0:1787758395.585750:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00000001:1.0:1787758395.585757:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758395.585760:0:2357:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758395.585764:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758395.585765:0:2357:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758395.585768:0:2357:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758395.585770:0:2357:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:1.0:1787758395.585781:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758392/real 1787758395] req@ffff8ffb4790e680 x1874588659743488/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758395.585799:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758395.585802:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758395.585810:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758395.585814:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758395.585816:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.585820:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758395.585823:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758395.585829:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790e680 x1874588659743488/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758395.585845:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.585850:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758395.585853:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.585855:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758395.585856:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.585860:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758395.585882:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758395.585886:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758395.585893:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758395.585897:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758395.585902:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743488:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758395.585908:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758395.585912:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758392/real 1787758395] req@ffff8ffb4790d180 x1874588659743616/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758395.585923:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758395.585924:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758395.585928:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758395.585930:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758395.585931:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.585934:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758395.585935:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758395.585940:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790d180 x1874588659743616/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758395.585947:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.585949:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758395.585952:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.585953:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758395.585954:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.585956:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758395.585974:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758395.585976:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758395.585980:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758395.585983:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758395.585987:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743616:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758395.585992:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758395.585997:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758392/real 1787758395] req@ffff8ffb4790ca80 x1874588659743360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758395.586004:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758395.586006:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758395.586009:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758395.586011:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758395.586013:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586014:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758395.586016:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758395.586019:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ca80 x1874588659743360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758395.586025:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586036:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758395.586038:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586040:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758395.586041:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586043:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758395.586045:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758395.586047:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758395.586050:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758395.586053:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758395.586056:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659743360:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758395.586063:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758395.586066:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758395.586068:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758395.586072:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758395.586081:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758395.586086:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758395.586089:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758395.586095:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758395.586098:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758395.586103:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758395.586104:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586106:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758395.586108:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758395.586109:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758395.586114:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758395.586115:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758395.586116:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758395.586119:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758395.586120:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758395.586122:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758395.586123:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586125:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758395.586126:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758395.586128:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758395.586130:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758395.586131:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758395.586132:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758395.586134:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758395.586135:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758395.586137:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758395.586138:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586141:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758395.586148:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758395.586150:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758395.586152:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758395.586157:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758392/real 1787758395] req@ffff8ffb4790fb80 x1874588659743744/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758395.586167:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758395.586168:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758395.586177:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758395.586179:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758395.586215:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586218:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758395.586219:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758395.586223:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790fb80 x1874588659743744/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758395.586230:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586232:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758395.586235:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586236:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758395.586238:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586240:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758395.586269:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758395.586273:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758395.586277:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758395.586279:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758395.586282:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659743744:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758395.586286:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586288:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758395.586289:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758395.586291:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758395.586293:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758395.586295:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758395.586297:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758395.586301:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758395.586303:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758395.586305:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758395.586306:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586309:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758395.586321:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758395.586324:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758395.586327:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758395.586329:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758396.161228:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758396.161242:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758396.161246:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758396.161251:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758396.161255:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758396.161260:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758396.161262:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758396.161265:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758396.161267:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758396.353309:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758396.353326:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758396.353333:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758396.353334:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758396.353356:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758396.353358:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758396.353361:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758397.185241:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758397.185250:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758397.185252:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758397.185257:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758397.185264:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758397.185269:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758397.185272:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758397.185276:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758397.185279:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758397.377337:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758397.377348:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758397.377352:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758397.377356:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758397.377363:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758397.377367:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758397.377369:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758397.377372:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758397.377377:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758397.377379:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758397.377465:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758397.377471:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758397.377476:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758397.633342:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758397.633354:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:0.0:1787758397.633357:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758397.633361:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758397.633366:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758397.633374:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11452 00000100:00080000:0.0:1787758397.633387:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758397.633391:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758397.633393:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758397.633398:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758397.633401:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758397.633404:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:0.0:1787758397.633407:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:0.0:1787758397.633410:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758397.633412:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758397.633417:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758397.633421:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758397.633424:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758397.633428:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758397.633432:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633434:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:0.0:1787758397.633437:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758397.633447:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758397.633449:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633454:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758397.633457:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758397.633462:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633475:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1787758397.633486:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ed80 x1874588659743872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:0.0:1787758397.633499:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:0.0:1787758397.633504:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633524:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758397.633529:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758397.633533:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:0.0:1787758397.633535:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:0.0:1787758397.633537:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:0.0:1787758397.633540:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:0.0:1787758397.633543:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11452 00000100:00080000:0.0:1787758397.633546:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758397.633548:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758397.633550:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758397.633553:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758397.633554:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758397.633557:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:0.0:1787758397.633559:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:0.0:1787758397.633562:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758397.633564:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758397.633567:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758397.633570:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758397.633573:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758397.633578:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758397.633580:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633582:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758397.633614:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758397.633620:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633625:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758397.633632:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758397.633642:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633645:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758397.633648:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758397.633652:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:0.0:1787758397.633653:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.633665:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1787758397.633669:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.633673:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:0.0:1787758397.633685:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790ce00 x1874588659744000/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758397.633689:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758397.633696:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.633708:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:0.0:1787758397.633714:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1787758397.633716:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1787758397.633725:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.633728:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633736:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758397.633744:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659743872:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1787758397.633746:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758397.633763:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:0.0:1787758397.633764:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:0.0:1787758397.633772:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:1.0:1787758397.633776:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00080000:0.0:1787758397.633777:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787758397.633784:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633787:0:87148:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787758397.633795:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:0.0:1787758397.633802:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11452 02000000:00000001:1.0:1787758397.633809:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1787758397.633814:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:0.0:1787758397.633821:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758397.633824:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758397.633830:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758397.633832:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758397.633837:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758397.633838:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000020:00000001:0.0:1787758397.633844:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758397.633846:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133084544 : -123165576467072 : ffff8ffb4790ed80) 00000100:00000001:0.0:1787758397.633858:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758397.633861:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:0.0:1787758397.633865:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:0.0:1787758397.633876:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:0.0:1787758397.633881:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1787758397.633885:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1787758397.633892:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1787758397.633896:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633901:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758397.633906:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:0.0:1787758397.633910:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633914:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758397.633923:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:0.0:1787758397.633927:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758397.633933:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.633938:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:0.0:1787758397.633942:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758397.633944:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:0.0:1787758397.633947:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.633955:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:0.0:1787758397.633962:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.633963:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.633974:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00100000:0.0:1787758397.633986:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790f100 x1874588659744128/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758397.633986:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.633998:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306125505 00000100:00000001:0.0:1787758397.634015:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758397.634019:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:0.0:1787758397.634021:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.634031:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634034:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306125505 00000100:00080000:0.0:1787758397.634037:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758397.634047:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:0.0:1787758397.634049:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:0.0:1787758397.634058:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:1.0:1787758397.634059:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:0.0:1787758397.634062:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758397.634072:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1787758397.634074:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758397.634082:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00080000:0.0:1787758397.634086:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11452 00000100:00080000:0.0:1787758397.634097:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758397.634100:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758397/real 1787758397] req@ffff8ffb4790ed80 x1874588659743872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1787758397.634103:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758397.634109:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:0.0:1787758397.634119:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:0.0:1787758397.634125:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:0.0:1787758397.634131:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787758397.634138:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:0.0:1787758397.634139:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:1.0:1787758397.634145:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1787758397.634147:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:0.0:1787758397.634154:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758397.634158:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00080000:0.0:1787758397.634164:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758397.634168:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1787758397.634172:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634175:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.634226:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758397.634230:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758397.634243:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ed80 x1874588659743872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:0.0:1787758397.634254:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758397.634263:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:0.0:1787758397.634267:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758397.634273:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:0.0:1787758397.634276:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634286:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1787758397.634287:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:0.0:1787758397.634295:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758397.634296:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758397.634301:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000008:00000001:0.0:1787758397.634307:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634310:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:0.0:1787758397.634316:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634318:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758397.634329:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 02000000:00000001:0.0:1787758397.634331:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:0.0:1787758397.634337:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758397.634340:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:0.0:1787758397.634345:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758397.634350:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:0.0:1787758397.634352:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:0.0:1787758397.634360:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758397.634365:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659743872:192.168.201.118@tcp:250:kworker.0 00000100:00000001:0.0:1787758397.634374:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634378:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.634386:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758397.634392:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00100000:0.0:1787758397.634394:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4790c380 x1874588659744256/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758397.634398:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758397.634409:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758397.634415:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000100:00000001:0.0:1787758397.634418:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000020:00000001:1.0:1787758397.634421:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:0.0:1787758397.634428:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.634433:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:0.0:1787758397.634437:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.634440:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758397.634450:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758397.634453:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.634460:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.634471:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758397.634475:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634481:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758397.634484:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758397.634487:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758397.634491:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758397.634494:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.634498:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758397.634502:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758397.634506:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758397.634514:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744000:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758397.634520:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758397.634525:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758397.634529:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.634533:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758397.634540:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634554:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758397.634558:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133076480 : -123165576475136 : ffff8ffb4790ce00) 00000100:00000001:1.0:1787758397.634566:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758397.634625:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758397.634636:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758397.634642:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758397.634645:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634649:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758397.634656:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634663:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306124814 00000100:00000001:1.0:1787758397.634668:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306125505 00000100:00000001:1.0:1787758397.634672:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306124814 00000100:00000001:1.0:1787758397.634676:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306125505 00000100:00000001:1.0:1787758397.634680:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758397.634684:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634688:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758397.634697:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758397/real 1787758397] req@ffff8ffb4790ce00 x1874588659744000/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758397.634712:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758397.634715:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758397.634720:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758397.634723:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758397.634726:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.634729:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758397.634732:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758397.634738:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790ce00 x1874588659744000/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758397.634750:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.634754:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758397.634758:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.634761:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758397.634764:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.634767:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758397.634791:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758397.634797:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758397.634803:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758397.634807:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758397.634813:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744000:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758397.634820:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758397.634823:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758397.634825:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.634830:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758397.634833:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758397.634836:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758397.634842:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744128:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758397.634848:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758397.634852:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758397.634854:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.634857:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758397.634863:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634874:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758397.634877:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133085440 : -123165576466176 : ffff8ffb4790f100) 00000100:00000001:1.0:1787758397.634884:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758397.634911:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758397.634920:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758397.634924:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758397.634927:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634930:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758397.634936:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634942:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306124811 00000100:00000001:1.0:1787758397.634947:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306125505 00000100:00000001:1.0:1787758397.634951:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306124811 00000100:00000001:1.0:1787758397.634955:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306125505 00000100:00000001:1.0:1787758397.634959:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758397.634963:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.634966:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758397.634975:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758397/real 1787758397] req@ffff8ffb4790f100 x1874588659744128/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758397.634991:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758397.634993:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758397.634998:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758397.635001:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758397.635003:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635007:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758397.635009:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758397.635016:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790f100 x1874588659744128/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758397.635029:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635033:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758397.635038:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635041:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758397.635043:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635046:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758397.635069:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758397.635074:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758397.635081:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758397.635086:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758397.635091:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744128:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758397.635098:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758397.635101:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758397.635104:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.635108:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758397.635111:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758397.635115:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758397.635121:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758397.635127:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758397.635131:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758397.635134:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758397.635137:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758397.635142:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.635152:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758397.635155:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908133073792 : -123165576477824 : ffff8ffb4790c380) 00000100:00000001:1.0:1787758397.635162:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758397.635225:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758397.635235:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758397.635240:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758397.635243:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.635246:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758397.635251:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.635258:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306124812 00000100:00000001:1.0:1787758397.635263:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306125505 00000100:00000001:1.0:1787758397.635267:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306124812 00000100:00000001:1.0:1787758397.635271:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306125505 00000100:00000001:1.0:1787758397.635275:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758397.635279:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.635282:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758397.635291:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758397/real 1787758397] req@ffff8ffb4790c380 x1874588659744256/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758397.635307:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758397.635310:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758397.635315:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758397.635318:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758397.635321:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635324:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758397.635326:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758397.635333:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4790c380 x1874588659744256/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758397.635346:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635350:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758397.635355:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635357:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758397.635360:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635363:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758397.635383:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758397.635388:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758397.635394:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758397.635399:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758397.635405:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4790c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744256:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758397.635412:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635416:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758397.635419:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758397.635422:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758397.635427:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758397.635429:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758397.635431:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758397.635434:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758397.635436:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758397.635439:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758397.635441:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635443:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758397.635444:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758397.635446:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758397.635448:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758397.635450:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758397.635451:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758397.635454:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758397.635455:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758397.635457:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758397.635458:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635461:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758397.635463:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758397.635465:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758397.635468:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758397.635470:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758397.635472:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758397.635475:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758397.635477:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758397.635480:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758397.635482:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635486:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758397.635494:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758397.635496:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758397.635499:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758397.635502:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758398.209243:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758398.209249:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758398.209251:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758398.209255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758398.209259:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758398.209263:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758398.209265:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758398.209267:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758398.209269:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758398.401284:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758398.401290:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758398.401295:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758398.401297:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758398.401312:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758398.401314:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758398.401316:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758399.233264:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758399.233271:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758399.233273:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758399.233279:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758399.233282:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758399.233286:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758399.233288:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758399.233291:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758399.233294:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758399.425355:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758399.425362:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758399.425381:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758399.425383:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758399.425473:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758399.425478:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758399.425482:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758400.257259:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758400.257267:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758400.257269:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758400.257275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758400.257279:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758400.257282:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758400.257284:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758400.257285:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758400.257288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758400.449294:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758400.449301:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758400.449308:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758400.449311:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758400.449333:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758400.449336:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758400.449339:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758401.281273:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758401.281282:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758401.281284:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758401.281290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758401.281293:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758401.281297:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758401.281300:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758401.281303:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758401.281305:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758401.473301:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758401.473308:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758401.473314:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758401.473315:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758401.473396:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758401.473400:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758401.473402:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758402.305262:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758402.305270:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758402.305272:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758402.305278:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758402.305281:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758402.305285:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758402.305287:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758402.305289:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758402.305292:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758402.497289:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758402.497295:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758402.497299:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758402.497299:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758402.497314:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758402.497324:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758402.497327:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758402.753296:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758402.753307:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758402.753310:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758402.753314:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758402.753318:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758402.753326:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11457 00000100:00080000:3.0:1787758402.753335:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758402.753338:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758402.753339:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758402.753342:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758402.753344:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758402.753347:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758402.753349:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758402.753350:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758402.753351:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758402.753355:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758402.753356:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753358:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758402.753361:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758402.753363:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753365:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758402.753367:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753377:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758402.753378:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753381:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758402.753383:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758402.753388:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753398:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758402.753405:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659744384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758402.753413:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758402.753426:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753438:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758402.753440:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758402.753443:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758402.753444:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758402.753448:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758402.753451:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758402.753452:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11457 00000100:00080000:3.0:1787758402.753455:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758402.753456:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758402.753458:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758402.753459:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758402.753460:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758402.753462:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758402.753464:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758402.753467:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758402.753468:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758402.753472:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758402.753474:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753476:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758402.753481:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758402.753484:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753486:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758402.753491:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758402.753509:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753511:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753516:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758402.753518:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753520:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758402.753521:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758402.753523:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753529:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1787758402.753529:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753536:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00100000:3.0:1787758402.753538:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659744512/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758402.753541:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758402.753546:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758402.753548:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758402.753550:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753551:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753557:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758402.753559:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:3.0:1787758402.753560:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787758402.753565:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787758402.753565:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758402.753569:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758402.753572:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 02000000:00000001:1.0:1787758402.753573:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753575:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758402.753580:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11457 00000100:00100000:1.0:1787758402.753583:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659744384:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1787758402.753584:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758402.753588:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758402.753590:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758402.753593:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:1.0:1787758402.753594:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:3.0:1787758402.753595:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758402.753599:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758402.753601:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:3.0:1787758402.753601:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758402.753605:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:1.0:1787758402.753606:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753607:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758402.753613:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00080000:3.0:1787758402.753614:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758402.753617:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753618:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758402.753620:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753621:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758402.753626:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753628:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758402.753630:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758402.753632:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753634:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753637:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758402.753638:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753640:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758402.753641:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:3.0:1787758402.753641:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758402.753646:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 02000000:00000001:3.0:1787758402.753646:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753652:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753654:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00100000:3.0:1787758402.753658:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659744640/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758402.753664:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758402.753665:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753667:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758402.753669:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758402.753671:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:3.0:1787758402.753672:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758402.753673:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758402.753675:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758402.753676:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11457 00000100:00080000:3.0:1787758402.753679:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758402.753680:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758402.753681:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758402.753682:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758402.753683:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758402.753684:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758402.753686:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:3.0:1787758402.753688:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758402.753688:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758402.753691:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758402.753693:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753694:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758402.753696:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000800:00000100:1.0:1787758402.753700:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 02000000:00000001:3.0:1787758402.753701:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753704:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758402.753706:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758402.753708:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753709:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758402.753713:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758402.753714:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753715:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758402.753716:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:1.0:1787758402.753717:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787758402.753717:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753723:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753725:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00100000:3.0:1787758402.753728:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659744768/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758402.753729:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753735:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758402.753740:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758402.753742:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753743:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758402.753744:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753753:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306130625 00000100:00000001:1.0:1787758402.753765:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758402.753771:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306130625 00000100:00000001:1.0:1787758402.753775:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758402.753778:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758402.753781:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753784:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758402.753790:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758402/real 1787758402] req@ffff8ffb45c30000 x1874588659744384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758402.753801:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758402.753803:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758402.753807:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758402.753809:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758402.753811:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.753813:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758402.753815:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758402.753820:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659744384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758402.753828:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.753831:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758402.753835:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.753836:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758402.753838:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.753840:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758402.753842:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758402.753845:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.753848:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758402.753851:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758402.753855:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659744384:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758402.753859:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.753861:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758402.753863:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758402.753865:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758402.753869:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758402.753871:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758402.753873:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758402.753876:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758402.753878:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758402.753881:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758402.753882:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.753884:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.753892:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758402.753894:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753896:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758402.753898:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758402.753899:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758402.753901:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758402.753902:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758402.753905:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758402.753907:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758402.753909:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758402.753912:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744512:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758402.753915:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758402.753917:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758402.753919:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758402.753921:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758402.753923:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753938:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758402.753939:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:1.0:1787758402.753945:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758402.753958:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758402.753962:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758402.753965:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758402.753966:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753967:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758402.753970:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753974:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306129933 00000100:00000001:1.0:1787758402.753976:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306130625 00000100:00000001:1.0:1787758402.753978:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306129933 00000100:00000001:1.0:1787758402.753980:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306130625 00000100:00000001:1.0:1787758402.753983:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758402.753984:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.753986:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758402.753990:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758402/real 1787758402] req@ffff8ffb45c31180 x1874588659744512/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758402.753998:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758402.753999:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758402.754001:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758402.754002:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758402.754004:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754006:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758402.754007:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758402.754010:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659744512/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758402.754017:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754019:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758402.754022:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754024:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758402.754025:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754027:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758402.754040:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758402.754043:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.754046:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758402.754048:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758402.754052:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744512:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758402.754055:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758402.754057:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758402.754058:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758402.754060:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758402.754061:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758402.754064:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758402.754066:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744640:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758402.754070:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758402.754071:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758402.754073:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758402.754074:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758402.754077:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754082:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758402.754083:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:1.0:1787758402.754087:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758402.754098:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758402.754103:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758402.754105:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758402.754106:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754108:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758402.754111:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754114:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306129930 00000100:00000001:1.0:1787758402.754117:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306130625 00000100:00000001:1.0:1787758402.754119:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306129930 00000100:00000001:1.0:1787758402.754132:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306130625 00000100:00000001:1.0:1787758402.754134:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758402.754136:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754139:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758402.754143:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758402/real 1787758402] req@ffff8ffb45c33480 x1874588659744640/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758402.754152:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758402.754154:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758402.754157:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758402.754158:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758402.754160:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754162:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758402.754163:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758402.754167:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659744640/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758402.754175:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754177:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758402.754179:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754181:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758402.754182:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754212:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758402.754225:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758402.754229:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.754233:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758402.754235:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758402.754238:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744640:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758402.754242:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758402.754244:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758402.754245:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758402.754247:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758402.754249:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758402.754251:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758402.754254:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758402.754260:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758402.754262:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758402.754264:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758402.754266:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758402.754269:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754275:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758402.754276:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00000001:1.0:1787758402.754280:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758402.754293:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758402.754298:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758402.754300:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758402.754302:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754303:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758402.754306:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754310:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306129931 00000100:00000001:1.0:1787758402.754313:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306130625 00000100:00000001:1.0:1787758402.754315:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306129931 00000100:00000001:1.0:1787758402.754317:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306130625 00000100:00000001:1.0:1787758402.754319:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758402.754321:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754323:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758402.754328:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758402/real 1787758402] req@ffff8ffb45c33800 x1874588659744768/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758402.754336:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758402.754337:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758402.754340:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758402.754341:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758402.754343:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754345:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758402.754346:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758402.754351:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659744768/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758402.754364:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754367:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758402.754369:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754370:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758402.754372:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754374:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758402.754395:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758402.754398:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.754401:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758402.754403:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758402.754407:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659744768:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758402.754411:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754413:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758402.754414:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758402.754416:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758402.754419:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758402.754420:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758402.754422:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758402.754424:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758402.754425:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758402.754427:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758402.754428:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754430:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758402.754431:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758402.754432:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758402.754434:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758402.754436:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758402.754437:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758402.754438:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758402.754440:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758402.754441:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758402.754442:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754444:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758402.754445:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758402.754447:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758402.754449:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758402.754451:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758402.754452:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758402.754453:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758402.754454:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758402.754456:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758402.754457:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754459:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758402.754463:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758402.754464:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758402.754467:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758402.754468:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758403.329279:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758403.329286:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758403.329288:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758403.329291:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758403.329294:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758403.329297:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758403.329299:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758403.329301:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758403.329303:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758403.521299:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758403.521307:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758403.521314:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758403.521315:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758403.521358:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758403.521363:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758403.521367:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758404.353245:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758404.353251:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758404.353253:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758404.353256:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758404.353259:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758404.353262:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758404.353264:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758404.353266:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758404.353269:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758404.545279:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758404.545284:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758404.545288:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758404.545289:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758404.545305:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758404.545307:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758404.545309:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758405.377243:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758405.377250:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758405.377252:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758405.377256:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758405.377259:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758405.377263:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758405.377265:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758405.377267:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758405.377268:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758405.569341:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758405.569347:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758405.569353:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758405.569357:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758405.569359:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758405.569363:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758405.569366:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758405.569369:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758405.569370:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758405.569374:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758405.569466:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758405.569471:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758405.569476:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758406.401272:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758406.401283:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758406.401286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758406.401293:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758406.401297:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758406.401302:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758406.401305:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758406.401309:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758406.401312:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758406.593286:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758406.593296:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758406.593303:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758406.593304:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758406.593333:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758406.593336:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758406.593340:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758407.425256:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758407.425265:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758407.425267:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758407.425272:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758407.425276:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758407.425280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758407.425282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758407.425285:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758407.425288:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758407.617298:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758407.617309:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758407.617312:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758407.617315:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758407.617321:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758407.617323:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758407.617325:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758407.617327:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758407.617332:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758407.617334:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758407.617400:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758407.617404:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758407.617407:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758407.873339:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758407.873348:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758407.873349:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758407.873352:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758407.873357:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758407.873363:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11462 00000100:00080000:2.0:1787758407.873372:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758407.873374:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758407.873376:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758407.873379:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758407.873381:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758407.873383:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758407.873385:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758407.873386:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758407.873387:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758407.873390:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758407.873392:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758407.873394:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758407.873397:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758407.873399:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873401:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758407.873417:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758407.873428:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758407.873430:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873432:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758407.873434:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758407.873438:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873447:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758407.873455:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659744896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758407.873463:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758407.873466:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873478:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758407.873481:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758407.873483:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758407.873484:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758407.873486:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758407.873488:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758407.873490:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11462 00000100:00080000:2.0:1787758407.873492:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758407.873494:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758407.873495:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758407.873496:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758407.873497:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758407.873499:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758407.873500:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758407.873502:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758407.873502:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758407.873504:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758407.873506:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758407.873507:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758407.873511:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758407.873513:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873514:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758407.873518:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758407.873520:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873522:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758407.873525:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758407.873526:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873527:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758407.873528:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758407.873532:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758407.873533:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873535:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758407.873537:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873539:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00100000:2.0:1787758407.873543:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659745024/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758407.873545:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758407.873549:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873553:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758407.873555:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.873557:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1787758407.873559:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.873562:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00080000:2.0:1787758407.873563:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787758407.873570:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758407.873570:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758407.873572:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758407.873575:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758407.873577:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:1.0:1787758407.873582:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659744896:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787758407.873582:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11462 00000100:00080000:2.0:1787758407.873587:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758407.873590:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758407.873593:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758407.873594:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000020:00000001:2.0:1787758407.873596:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758407.873598:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758407.873599:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:1.0:1787758407.873600:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000020:00000001:2.0:1787758407.873601:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758407.873604:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758407.873605:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758407.873606:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758407.873608:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758407.873613:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.873614:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758407.873614:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758407.873619:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758407.873620:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.873622:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873623:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758407.873626:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758407.873628:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873629:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758407.873633:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758407.873633:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873634:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758407.873636:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758407.873637:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873641:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758407.873641:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873646:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00100000:2.0:1787758407.873648:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659745152/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758407.873656:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758407.873662:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758407.873663:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873665:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758407.873666:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758407.873669:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758407.873669:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758407.873670:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758407.873672:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758407.873674:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11462 00000100:00080000:2.0:1787758407.873676:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758407.873677:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758407.873678:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758407.873679:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758407.873681:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758407.873682:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758407.873684:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758407.873685:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758407.873686:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758407.873688:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758407.873690:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758407.873691:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758407.873693:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758407.873694:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873696:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758407.873699:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758407.873701:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00000001:2.0:1787758407.873701:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873704:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758407.873708:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758407.873709:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873710:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758407.873711:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758407.873713:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873716:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873717:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00100000:2.0:1787758407.873720:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659745280/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758407.873724:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758407.873727:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758407.873729:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873730:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758407.873734:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873736:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758407.873744:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873749:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306135745 00000100:00000001:1.0:1787758407.873759:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758407.873766:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306135745 00000100:00000001:1.0:1787758407.873770:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758407.873773:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758407.873776:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873779:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758407.873786:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758407/real 1787758407] req@ffff8ffb47a38380 x1874588659744896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758407.873797:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758407.873799:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758407.873803:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758407.873806:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758407.873807:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.873810:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758407.873812:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758407.873817:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659744896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758407.873825:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.873828:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758407.873831:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.873833:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758407.873834:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.873836:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758407.873838:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758407.873842:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758407.873846:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758407.873849:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758407.873853:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659744896:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758407.873858:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.873860:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758407.873862:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758407.873864:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758407.873868:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758407.873870:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758407.873872:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758407.873876:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758407.873878:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758407.873881:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758407.873882:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.873884:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.873891:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758407.873893:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873895:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758407.873897:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758407.873898:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758407.873900:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758407.873901:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.873904:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758407.873906:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758407.873908:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758407.873912:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745024:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758407.873915:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758407.873917:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758407.873919:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.873921:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758407.873924:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873929:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758407.873931:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:1.0:1787758407.873935:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758407.873947:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758407.873952:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758407.873954:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758407.873956:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873958:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758407.873960:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873964:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306135053 00000100:00000001:1.0:1787758407.873967:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306135745 00000100:00000001:1.0:1787758407.873969:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306135053 00000100:00000001:1.0:1787758407.873972:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306135745 00000100:00000001:1.0:1787758407.873974:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758407.873976:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.873978:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758407.873983:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758407/real 1787758407] req@ffff8ffb47a38000 x1874588659745024/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758407.873992:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758407.873993:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758407.873995:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758407.873997:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758407.873998:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874000:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758407.874001:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758407.874005:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659745024/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758407.874013:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874015:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758407.874018:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874020:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758407.874021:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874023:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758407.874036:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758407.874039:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758407.874042:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758407.874044:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758407.874048:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745024:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758407.874052:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758407.874053:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758407.874054:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.874056:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758407.874058:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758407.874060:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758407.874064:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745152:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758407.874067:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758407.874069:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758407.874070:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.874072:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758407.874075:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874080:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758407.874081:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787758407.874085:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758407.874096:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758407.874100:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758407.874102:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758407.874103:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874105:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758407.874108:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874111:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306135050 00000100:00000001:1.0:1787758407.874114:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306135745 00000100:00000001:1.0:1787758407.874116:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306135050 00000100:00000001:1.0:1787758407.874119:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306135745 00000100:00000001:1.0:1787758407.874121:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758407.874123:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874125:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758407.874129:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758407/real 1787758407] req@ffff8ffb47a3b480 x1874588659745152/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758407.874137:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758407.874139:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758407.874141:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758407.874142:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758407.874144:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874146:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758407.874147:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758407.874151:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659745152/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758407.874164:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874167:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758407.874169:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874171:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758407.874172:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874174:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758407.874217:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758407.874220:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758407.874224:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758407.874226:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758407.874230:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745152:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758407.874235:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758407.874236:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758407.874238:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.874240:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758407.874242:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758407.874245:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758407.874248:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758407.874251:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758407.874253:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758407.874255:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758407.874256:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758407.874259:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874264:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758407.874266:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787758407.874270:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758407.874281:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758407.874286:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758407.874288:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758407.874290:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874292:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758407.874294:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874298:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306135051 00000100:00000001:1.0:1787758407.874300:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306135745 00000100:00000001:1.0:1787758407.874303:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306135051 00000100:00000001:1.0:1787758407.874305:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306135745 00000100:00000001:1.0:1787758407.874308:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758407.874310:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874312:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758407.874316:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758407/real 1787758407] req@ffff8ffb47a39180 x1874588659745280/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758407.874324:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758407.874326:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758407.874328:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758407.874329:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758407.874331:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874332:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758407.874334:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758407.874338:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659745280/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758407.874345:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874348:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758407.874350:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874352:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758407.874353:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874355:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758407.874367:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758407.874370:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758407.874374:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758407.874376:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758407.874381:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745280:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758407.874385:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874387:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758407.874388:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758407.874390:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758407.874393:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758407.874395:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758407.874396:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758407.874399:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758407.874400:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758407.874419:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758407.874421:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874423:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758407.874425:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758407.874426:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758407.874428:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758407.874429:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758407.874430:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758407.874434:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758407.874435:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758407.874438:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758407.874439:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874441:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758407.874442:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758407.874443:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758407.874445:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758407.874446:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758407.874448:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758407.874449:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758407.874451:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758407.874454:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758407.874455:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874457:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758407.874461:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758407.874462:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758407.874464:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758407.874466:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758408.449264:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758408.449270:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758408.449271:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758408.449275:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758408.449278:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758408.449280:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758408.449282:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758408.449285:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758408.449287:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758408.641342:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758408.641350:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758408.641355:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758408.641356:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758408.641376:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758408.641379:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758408.641381:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758409.473332:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758409.473340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758409.473342:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758409.473348:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758409.473351:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758409.473355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758409.473357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758409.473359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758409.473362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758409.665283:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758409.665291:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758409.665299:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758409.665300:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758409.665351:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758409.665355:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758409.665358:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758410.497236:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758410.497243:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758410.497245:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758410.497248:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758410.497251:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758410.497254:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758410.497255:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758410.497257:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758410.497259:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758410.689282:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758410.689289:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758410.689295:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758410.689296:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758410.689313:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758410.689315:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758410.689317:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758411.521245:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758411.521251:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758411.521254:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758411.521258:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758411.521261:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758411.521265:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758411.521266:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758411.521268:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758411.521270:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758411.713259:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758411.713265:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758411.713269:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758411.713270:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758411.713310:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758411.713314:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758411.713316:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:3.0:1787758412.545247:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758412.545253:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758412.545254:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758412.545258:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758412.545261:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758412.545264:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758412.545266:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758412.545268:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758412.545270:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758412.737261:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758412.737270:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758412.737276:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758412.737277:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758412.737299:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758412.737301:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758412.737304:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758412.993302:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758412.993345:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758412.993348:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758412.993352:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758412.993363:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758412.993371:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11467 00000100:00080000:2.0:1787758412.993383:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758412.993386:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758412.993388:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758412.993392:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758412.993395:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758412.993398:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758412.993401:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758412.993403:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758412.993405:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758412.993410:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758412.993413:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993416:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758412.993419:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758412.993422:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993425:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758412.993428:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993440:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758412.993442:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993447:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758412.993451:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758412.993455:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993466:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758412.993476:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38700 x1874588659745408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758412.993487:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758412.993492:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993508:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758412.993512:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758412.993516:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758412.993518:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758412.993520:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758412.993523:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758412.993525:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11467 00000100:00080000:2.0:1787758412.993530:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758412.993532:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758412.993534:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758412.993537:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758412.993539:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758412.993541:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758412.993544:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758412.993546:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758412.993548:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758412.993551:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00080000:2.0:1787758412.993552:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758412.993556:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758412.993557:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993559:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:2.0:1787758412.993561:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758412.993562:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758412.993565:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993568:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758412.993569:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758412.993572:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758412.993574:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.993575:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993580:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:1.0:1787758412.993584:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659745408:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1787758412.993587:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758412.993590:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:2.0:1787758412.993594:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.993595:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787758412.993597:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.993598:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.993603:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758412.993604:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758412.993607:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993608:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993613:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758412.993614:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758412.993617:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993623:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993625:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758412.993628:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134303488 : -123165575248128 : ffff8ffb47a38700) 00000100:00100000:2.0:1787758412.993631:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a300 x1874588659745536/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758412.993634:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:2.0:1787758412.993643:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758412.993646:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993649:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758412.993652:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758412.993656:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758412.993657:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758412.993659:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758412.993663:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758412.993667:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11467 00000100:00080000:2.0:1787758412.993671:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758412.993673:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000800:00000100:1.0:1787758412.993677:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758412.993677:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758412.993682:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758412.993684:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758412.993686:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758412.993689:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758412.993692:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758412.993692:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758412.993694:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758412.993697:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758412.993701:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758412.993702:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758412.993704:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758412.993707:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993710:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758412.993712:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993714:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758412.993717:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306140865 02000000:00000001:2.0:1787758412.993720:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993724:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758412.993727:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000008:00100000:2.0:1787758412.993728:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758412.993733:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993735:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306140865 00000100:00000001:2.0:1787758412.993736:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993740:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758412.993744:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758412.993745:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758412.993746:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993749:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993750:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:2.0:1787758412.993754:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00100000:1.0:1787758412.993757:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758412/real 1787758412] req@ffff8ffb47a38700 x1874588659745408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1787758412.993757:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758412.993761:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993770:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1787758412.993771:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758412.993774:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758412.993778:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758412.993781:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00100000:2.0:1787758412.993782:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39180 x1874588659745664/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758412.993783:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.993787:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758412.993790:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758412.993797:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38700 x1874588659745408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758412.993801:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758412.993807:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758412.993808:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993812:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758412.993814:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993815:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.993819:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00080000:2.0:1787758412.993820:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1787758412.993821:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.993824:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758412.993827:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758412.993828:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758412.993831:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:2.0:1787758412.993832:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758412.993836:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758412.993838:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758412.993841:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:1.0:1787758412.993843:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00080000:2.0:1787758412.993850:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11467 00000100:00100000:1.0:1787758412.993852:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659745408:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1787758412.993858:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758412.993859:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758412.993862:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758412.993863:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758412.993868:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 00000100:00000001:2.0:1787758412.993868:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:1.0:1787758412.993871:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 00000020:00000001:2.0:1787758412.993875:0:18724:0:(genops.c:792:class_conn2export()) Process entered 02000000:00000001:1.0:1787758412.993877:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758412.993878:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758412.993880:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758412.993883:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000020:00000001:2.0:1787758412.993884:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 02000000:00000001:1.0:1787758412.993887:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758412.993890:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000020:00000001:2.0:1787758412.993890:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758412.993894:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758412.993895:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758412.993896:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.993899:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758412.993899:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758412.993907:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00080000:2.0:1787758412.993907:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758412.993910:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993912:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993913:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758412.993915:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 02000000:00000001:2.0:1787758412.993918:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758412.993919:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758412.993923:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758412.993925:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993925:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758412.993928:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758412.993930:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.993931:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758412.993935:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993936:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:1.0:1787758412.993939:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745536:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1787758412.993942:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:1.0:1787758412.993944:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758412.993947:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000008:00000001:2.0:1787758412.993949:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.993950:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.993956:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758412.993957:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.993959:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758412.993963:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:1.0:1787758412.993965:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:2.0:1787758412.993967:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.993968:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134310656 : -123165575240960 : ffff8ffb47a3a300) 00000100:00000001:2.0:1787758412.993971:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:1.0:1787758412.993974:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787758412.993974:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758412.993978:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758412.993985:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758412.993991:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3b480 x1874588659745792/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758412.993998:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758412.994003:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758412.994006:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758412.994007:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758412.994011:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994013:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758412.994013:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994016:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994021:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306140173 00000100:00000001:2.0:1787758412.994021:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994026:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306140865 00000100:00000001:1.0:1787758412.994029:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306140173 00000100:00000001:1.0:1787758412.994031:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306140865 00000100:00000001:1.0:1787758412.994033:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758412.994034:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994036:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758412.994039:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758412/real 1787758412] req@ffff8ffb47a3a300 x1874588659745536/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758412.994045:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758412.994046:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758412.994047:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758412.994049:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758412.994051:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994052:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758412.994053:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758412.994056:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a300 x1874588659745536/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758412.994061:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994063:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758412.994064:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994065:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758412.994066:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994067:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758412.994077:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758412.994078:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758412.994080:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758412.994082:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758412.994084:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745536:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758412.994086:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758412.994087:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758412.994088:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.994090:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758412.994091:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758412.994092:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758412.994094:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745664:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758412.994096:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758412.994097:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758412.994098:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.994099:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758412.994101:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994104:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758412.994105:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134306176 : -123165575245440 : ffff8ffb47a39180) 00000100:00000001:1.0:1787758412.994108:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758412.994114:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758412.994117:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758412.994118:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758412.994119:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994120:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758412.994122:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994124:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306140170 00000100:00000001:1.0:1787758412.994126:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306140865 00000100:00000001:1.0:1787758412.994127:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306140170 00000100:00000001:1.0:1787758412.994129:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306140865 00000100:00000001:1.0:1787758412.994130:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758412.994131:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994132:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758412.994135:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758412/real 1787758412] req@ffff8ffb47a39180 x1874588659745664/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758412.994139:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758412.994140:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758412.994141:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758412.994142:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758412.994143:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994144:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758412.994144:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758412.994147:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39180 x1874588659745664/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758412.994151:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994152:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758412.994154:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994155:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758412.994156:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994157:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758412.994165:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758412.994166:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758412.994168:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758412.994169:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758412.994171:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745664:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758412.994174:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994175:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758412.994176:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758412.994177:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758412.994178:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758412.994179:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758412.994180:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758412.994217:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758412.994218:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758412.994221:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758412.994222:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994223:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758412.994224:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758412.994225:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758412.994226:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758412.994227:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758412.994228:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758412.994229:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758412.994230:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758412.994231:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758412.994232:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994233:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994235:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758412.994236:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994237:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758412.994238:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758412.994239:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758412.994240:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758412.994241:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.994243:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758412.994244:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758412.994245:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758412.994248:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745792:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758412.994250:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758412.994251:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758412.994252:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758412.994253:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758412.994255:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994258:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758412.994259:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134315136 : -123165575236480 : ffff8ffb47a3b480) 00000100:00000001:1.0:1787758412.994262:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758412.994270:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758412.994272:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758412.994274:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758412.994275:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994276:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758412.994277:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994279:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306140171 00000100:00000001:1.0:1787758412.994281:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306140865 00000100:00000001:1.0:1787758412.994283:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306140171 00000100:00000001:1.0:1787758412.994284:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306140865 00000100:00000001:1.0:1787758412.994286:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758412.994295:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994297:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758412.994302:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758412/real 1787758412] req@ffff8ffb47a3b480 x1874588659745792/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758412.994310:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758412.994311:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758412.994326:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758412.994329:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758412.994330:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994332:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758412.994333:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758412.994338:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3b480 x1874588659745792/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758412.994346:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994348:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758412.994351:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994353:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758412.994355:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994356:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758412.994365:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758412.994367:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758412.994369:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758412.994371:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758412.994373:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659745792:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758412.994376:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994377:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758412.994378:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758412.994379:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758412.994381:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758412.994382:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758412.994383:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758412.994385:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758412.994385:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758412.994387:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758412.994388:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994389:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758412.994391:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758412.994392:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758412.994394:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758412.994395:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758413.569269:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1787758413.569276:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1787758413.569278:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758413.569284:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:3.0:1787758413.569287:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758413.569291:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758413.569294:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758413.569296:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758413.569299:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758413.761250:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758413.761256:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758413.761261:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758413.761262:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758413.761315:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758413.761320:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758413.761324:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758414.593271:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758414.593280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758414.593282:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758414.593308:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758414.593313:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758414.593316:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758414.593319:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758414.593321:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758414.593324:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758414.785279:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758414.785299:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758414.785305:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758414.785306:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758414.785324:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758414.785327:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758414.785329:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758415.617243:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758415.617250:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758415.617251:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758415.617255:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758415.617258:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758415.617261:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758415.617263:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758415.617265:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758415.617291:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758415.809254:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758415.809260:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758415.809276:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758415.809280:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758415.809283:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758415.809286:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758415.809290:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758415.809293:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758415.809295:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758415.809299:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758415.809363:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758415.809366:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758415.809368:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758416.641350:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758416.641359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758416.641360:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758416.641365:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758416.641368:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758416.641372:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758416.641374:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758416.641377:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758416.641380:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758416.833382:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758416.833391:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758416.833399:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758416.833401:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758416.833429:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758416.833432:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758416.833436:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758417.665274:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758417.665280:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758417.665281:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758417.665285:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758417.665288:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758417.665292:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758417.665294:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758417.665296:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758417.665298:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758417.857290:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758417.857298:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758417.857301:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758417.857303:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758417.857308:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758417.857310:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758417.857312:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758417.857314:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758417.857318:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758417.857319:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758417.857341:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758417.857347:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758417.857352:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758418.113414:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758418.113427:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758418.113431:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758418.113435:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758418.113441:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758418.113449:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11472 00000100:00080000:3.0:1787758418.113461:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758418.113464:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758418.113466:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758418.113472:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758418.113475:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758418.113480:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758418.113485:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758418.113488:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758418.113490:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758418.113497:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758418.113500:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758418.113503:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758418.113508:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758418.113512:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.113515:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758418.113519:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758418.113545:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758418.113548:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.113552:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758418.113556:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758418.113561:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.113574:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758418.113586:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33800 x1874588659745920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758418.113599:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758418.113603:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.113628:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758418.113633:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758418.113638:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758418.113639:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758418.113642:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758418.113646:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758418.113651:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11472 00000100:00080000:3.0:1787758418.113657:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758418.113660:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758418.113663:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758418.113668:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758418.113669:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000001:3.0:1787758418.113670:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758418.113676:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758418.113678:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:3.0:1787758418.113680:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758418.113683:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1787758418.113685:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758418.113689:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:3.0:1787758418.113690:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758418.113694:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758418.113699:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787758418.113704:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:3.0:1787758418.113705:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.113712:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758418.113712:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758418.113721:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758418.113721:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758418.113730:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.113746:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:2.0:1787758418.113747:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659745920:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1787758418.113753:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758418.113756:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000008:00000001:3.0:1787758418.113759:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.113764:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758418.113765:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.113772:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758418.113775:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758418.113778:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758418.113787:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758418.113789:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.113794:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758418.113797:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758418.113799:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.113808:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1787758418.113809:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.113814:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825984 : -123165606725632 : ffff8ffb45c33800) 00000100:00100000:3.0:1787758418.113823:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c33480 x1874588659746048/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758418.113824:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758418.113872:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758418.113884:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758418.113888:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758418.113890:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.113893:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758418.113899:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.113904:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306145985 00000100:00000001:2.0:1787758418.113914:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:2.0:1787758418.113920:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306145985 00000100:00000001:2.0:1787758418.113925:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:3.0:1787758418.113933:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758418.113936:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.113950:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.113955:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787758418.113957:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787758418.113964:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758418.113966:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758418.113971:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758418.113972:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758418.113978:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758418.113986:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758418/real 1787758418] req@ffff8ffb45c33800 x1874588659745920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758418.113987:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758418.113997:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11472 00000100:00080000:3.0:1787758418.114006:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000400:00000001:2.0:1787758418.114022:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1787758418.114023:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:2.0:1787758418.114028:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758418.114029:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758418.114036:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000100:00000001:2.0:1787758418.114038:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000020:00000001:3.0:1787758418.114039:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758418.114046:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758418.114047:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000020:00000001:3.0:1787758418.114051:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:2.0:1787758418.114053:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758418.114056:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758418.114058:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758418.114061:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758418.114063:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758418.114071:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33800 x1874588659745920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758418.114087:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114091:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758418.114095:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114097:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758418.114098:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114101:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758418.114104:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758418.114107:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758418.114111:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758418.114114:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758418.114118:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659745920:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758418.114123:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114125:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758418.114127:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758418.114129:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758418.114134:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758418.114136:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758418.114138:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758418.114144:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758418.114146:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758418.114150:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758418.114151:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114153:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114161:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758418.114164:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114167:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758418.114169:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758418.114171:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758418.114172:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758418.114174:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.114177:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758418.114179:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758418.114182:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758418.114255:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746048:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758418.114266:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758418.114271:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758418.114276:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758418.114276:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 02000000:00000001:2.0:1787758418.114282:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758418.114288:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.114290:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758418.114298:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758418.114300:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758418.114305:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102825088 : -123165606726528 : ffff8ffb45c33480) 00000100:00000001:2.0:1787758418.114311:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758418.114330:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758418.114337:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758418.114339:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758418.114341:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114344:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758418.114350:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114356:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306145294 00000100:00000001:2.0:1787758418.114360:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306145985 00000100:00000001:2.0:1787758418.114364:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306145294 00000100:00000001:2.0:1787758418.114369:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306145985 02000000:00000001:3.0:1787758418.114371:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758418.114373:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758418.114382:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114383:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114392:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758418.114392:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758418.114398:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758418.114404:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758418/real 1787758418] req@ffff8ffb45c33480 x1874588659746048/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00000001:3.0:1787758418.114405:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.114412:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758418.114421:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000400:00000001:2.0:1787758418.114425:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 02000000:00000001:3.0:1787758418.114426:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:2.0:1787758418.114431:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:3.0:1787758418.114432:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758418.114437:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758418.114438:0:2367:0:(events.c:78:reply_in_callback()) Process entered 02000000:00000001:3.0:1787758418.114441:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114443:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758418.114448:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758418.114451:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114452:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758418.114457:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:3.0:1787758418.114468:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31180 x1874588659746176/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758418.114469:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c33480 x1874588659746048/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1787758418.114523:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758418.114530:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.114542:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114543:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758418.114553:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787758418.114554:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758418.114563:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758418.114566:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758418.114568:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758418.114572:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00000001:2.0:1787758418.114573:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758418.114577:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758418.114578:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:3.0:1787758418.114587:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758418.114593:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11472 00000100:00080000:3.0:1787758418.114600:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758418.114610:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758418.114613:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758418.114615:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1787758418.114616:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758418.114622:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00080000:2.0:1787758418.114625:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000020:00000001:3.0:1787758418.114627:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:3.0:1787758418.114634:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00080000:2.0:1787758418.114635:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:3.0:1787758418.114642:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758418.114644:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1787758418.114647:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787758418.114654:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c33480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746048:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1787758418.114660:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758418.114666:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758418.114667:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114673:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:3.0:1787758418.114674:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758418.114680:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:3.0:1787758418.114682:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758418.114684:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:3.0:1787758418.114690:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.114691:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787758418.114695:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000020:00000001:2.0:1787758418.114697:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000008:00100000:3.0:1787758418.114701:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000020:00000001:2.0:1787758418.114702:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758418.114708:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 00000008:00000001:3.0:1787758418.114709:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.114713:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:3.0:1787758418.114715:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114719:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 02000000:00000001:3.0:1787758418.114722:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758418.114724:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758418.114727:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114729:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758418.114733:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787758418.114737:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 02000000:00000001:3.0:1787758418.114738:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758418.114744:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758418.114745:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114752:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758418.114757:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758418.114757:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114763:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758418.114767:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758418.114773:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758418.114773:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30000 x1874588659746304/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787758418.114780:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758418.114786:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758418.114795:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758418.114797:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758418.114803:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758418.114807:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746176:192.168.201.118@tcp:8:kworker.0 00000100:00000001:3.0:1787758418.114810:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114819:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758418.114824:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758418.114827:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.114829:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758418.114834:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114841:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758418.114844:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102816128 : -123165606735488 : ffff8ffb45c31180) 00000100:00000001:2.0:1787758418.114849:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758418.114862:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758418.114867:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758418.114870:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758418.114872:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114874:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758418.114877:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114880:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306145291 00000100:00000001:2.0:1787758418.114883:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306145985 00000100:00000001:2.0:1787758418.114886:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306145291 00000100:00000001:2.0:1787758418.114888:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306145985 00000100:00000001:2.0:1787758418.114890:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758418.114892:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.114894:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758418.114898:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758418/real 1787758418] req@ffff8ffb45c31180 x1874588659746176/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758418.114907:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758418.114909:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758418.114911:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758418.114913:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758418.114914:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114916:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758418.114918:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758418.114922:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31180 x1874588659746176/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758418.114936:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114939:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758418.114952:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114954:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758418.114955:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114957:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758418.114968:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758418.114971:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758418.114974:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758418.114976:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758418.114979:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746176:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758418.114984:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.114985:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758418.114987:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758418.114989:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758418.114991:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758418.114992:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758418.114993:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758418.114995:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758418.114996:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758418.114999:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758418.115000:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115002:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115005:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758418.115007:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.115009:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758418.115010:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758418.115012:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758418.115014:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758418.115015:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.115018:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758418.115019:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758418.115022:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758418.115025:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746304:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758418.115028:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758418.115030:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758418.115032:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758418.115034:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758418.115037:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.115041:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758418.115042:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102811648 : -123165606739968 : ffff8ffb45c30000) 00000100:00000001:2.0:1787758418.115046:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758418.115057:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758418.115061:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758418.115063:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758418.115064:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.115066:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758418.115069:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.115072:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306145291 00000100:00000001:2.0:1787758418.115075:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306145985 00000100:00000001:2.0:1787758418.115077:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306145291 00000100:00000001:2.0:1787758418.115080:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306145985 00000100:00000001:2.0:1787758418.115082:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758418.115084:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.115086:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758418.115089:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758418/real 1787758418] req@ffff8ffb45c30000 x1874588659746304/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758418.115098:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758418.115099:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758418.115101:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758418.115103:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758418.115105:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115106:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758418.115108:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758418.115111:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30000 x1874588659746304/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758418.115125:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115128:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758418.115130:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115132:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758418.115133:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115135:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758418.115145:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758418.115147:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758418.115150:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758418.115152:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758418.115155:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746304:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758418.115159:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115161:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758418.115163:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758418.115164:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758418.115166:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758418.115167:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758418.115169:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758418.115170:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758418.115172:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758418.115173:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758418.115175:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115177:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758418.115179:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758418.115181:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758418.115210:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758418.115213:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758418.689369:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758418.689376:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758418.689378:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758418.689382:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758418.689385:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758418.689388:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758418.689390:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758418.689392:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758418.689394:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758418.881394:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758418.881400:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758418.881407:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758418.881409:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758418.881462:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758418.881472:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758418.881480:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758419.713329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758419.713338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758419.713340:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758419.713346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758419.713350:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758419.713354:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758419.713356:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758419.713359:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758419.713362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758419.905357:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758419.905367:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758419.905377:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758419.905379:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758419.905477:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758419.905483:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758419.905486:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758420.737370:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758420.737380:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758420.737383:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758420.737389:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758420.737394:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758420.737399:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758420.737403:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758420.737406:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758420.737410:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758420.929366:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758420.929379:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758420.929390:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758420.929392:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758420.929436:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758420.929439:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758420.929443:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758421.761329:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758421.761338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758421.761341:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758421.761346:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758421.761350:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758421.761355:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758421.761357:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758421.761360:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758421.761363:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758421.953355:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758421.953367:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758421.953378:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758421.953380:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758421.953417:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758421.953426:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758421.953432:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758422.785354:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758422.785364:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758422.785367:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758422.785374:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758422.785379:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758422.785384:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758422.785387:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758422.785390:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758422.785394:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758422.977322:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758422.977331:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758422.977338:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758422.977340:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758422.977431:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758422.977436:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758422.977441:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758423.233330:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758423.233340:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:3.0:1787758423.233343:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758423.233346:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758423.233357:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758423.233365:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11477 00000100:00080000:3.0:1787758423.233377:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758423.233380:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758423.233382:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758423.233386:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758423.233388:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758423.233391:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:3.0:1787758423.233394:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:3.0:1787758423.233396:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758423.233397:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758423.233402:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758423.233405:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758423.233407:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758423.233410:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758423.233413:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233415:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:3.0:1787758423.233418:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758423.233430:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758423.233432:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233435:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758423.233438:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:3.0:1787758423.233442:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233453:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1787758423.233479:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c31500 x1874588659746432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:3.0:1787758423.233494:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:3.0:1787758423.233499:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233527:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758423.233531:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758423.233536:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:3.0:1787758423.233538:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758423.233540:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758423.233544:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758423.233548:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11477 00000100:00080000:3.0:1787758423.233552:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758423.233555:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758423.233557:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758423.233560:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758423.233561:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758423.233564:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:3.0:1787758423.233568:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:3.0:1787758423.233570:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758423.233572:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758423.233577:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758423.233581:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758423.233583:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758423.233590:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758423.233593:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233596:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758423.233603:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000008:00100000:3.0:1787758423.233603:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758423.233612:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233613:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1787758423.233615:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233619:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 02000000:00000001:3.0:1787758423.233624:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758423.233626:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:3.0:1787758423.233629:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233632:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233634:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758423.233641:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758423.233644:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:3.0:1787758423.233645:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.233651:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1787758423.233655:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.233658:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758423.233675:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659746432:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1787758423.233676:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c32a00 x1874588659746560/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758423.233690:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758423.233698:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:3.0:1787758423.233701:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 02000000:00000001:2.0:1787758423.233703:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233708:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.233710:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1787758423.233719:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.233722:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1787758423.233725:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758423.233732:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:3.0:1787758423.233735:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758423.233737:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:3.0:1787758423.233741:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787758423.233745:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:3.0:1787758423.233746:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11477 00000100:00000001:2.0:1787758423.233751:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102817024 : -123165606734592 : ffff8ffb45c31500) 00000100:00080000:3.0:1787758423.233756:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758423.233760:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1787758423.233761:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758423.233766:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758423.233770:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758423.233772:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:3.0:1787758423.233775:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:3.0:1787758423.233777:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:3.0:1787758423.233780:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758423.233782:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:3.0:1787758423.233787:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:3.0:1787758423.233790:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758423.233792:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758423.233796:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1787758423.233810:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787758423.233831:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:3.0:1787758423.233833:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758423.233837:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758423.233839:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233842:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233845:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:3.0:1787758423.233847:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:3.0:1787758423.233850:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233851:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1787758423.233852:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:3.0:1787758423.233856:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758423.233857:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1787758423.233859:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233860:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:3.0:1787758423.233866:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233868:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233874:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306151105 00000100:00100000:3.0:1787758423.233875:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30700 x1874588659746688/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758423.233887:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:3.0:1787758423.233895:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758423.233896:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306151105 00000100:00000001:3.0:1787758423.233898:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233904:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233905:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:3.0:1787758423.233907:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1787758423.233910:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:3.0:1787758423.233912:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:2.0:1787758423.233915:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233916:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:3.0:1787758423.233921:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758423.233922:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:3.0:1787758423.233925:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:3.0:1787758423.233931:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11477 00000100:00100000:2.0:1787758423.233932:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758423/real 1787758423] req@ffff8ffb45c31500 x1874588659746432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758423.233935:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:3.0:1787758423.233941:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:3.0:1787758423.233944:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:3.0:1787758423.233951:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:3.0:1787758423.233954:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000400:00000001:2.0:1787758423.233955:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000020:00000001:3.0:1787758423.233956:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000400:00000001:2.0:1787758423.233958:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000020:00000001:3.0:1787758423.233960:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758423.233963:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1787758423.233965:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758423.233969:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:3.0:1787758423.233969:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758423.233973:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00080000:3.0:1787758423.233974:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758423.233979:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:3.0:1787758423.233980:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.233982:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758423.233984:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1787758423.233988:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00100000:2.0:1787758423.233990:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c31500 x1874588659746432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1787758423.233991:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.233997:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:3.0:1787758423.234001:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:3.0:1787758423.234009:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234010:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758423.234012:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234015:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787758423.234017:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758423.234019:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:3.0:1787758423.234021:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234022:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758423.234025:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1787758423.234027:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787758423.234029:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 02000000:00000001:3.0:1787758423.234031:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758423.234033:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:3.0:1787758423.234035:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758423.234039:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00000001:3.0:1787758423.234041:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758423.234045:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758423.234049:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1787758423.234051:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb45c30a80 x1874588659746816/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758423.234056:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c31500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659746432:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758423.234066:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234070:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758423.234072:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758423.234074:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758423.234077:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1787758423.234078:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000020:00000001:2.0:1787758423.234080:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758423.234084:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 00000100:00000001:3.0:1787758423.234084:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1787758423.234091:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.234092:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758423.234095:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758423.234098:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758423.234100:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234102:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234109:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758423.234111:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234114:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758423.234115:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758423.234117:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758423.234118:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758423.234120:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.234122:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758423.234124:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758423.234126:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758423.234129:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746560:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758423.234132:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758423.234179:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758423.234220:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.234223:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758423.234227:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234236:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758423.234238:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102822400 : -123165606729216 : ffff8ffb45c32a00) 00000100:00000001:2.0:1787758423.234244:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758423.234263:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758423.234269:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758423.234273:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758423.234275:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234277:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758423.234280:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234284:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306150414 00000100:00000001:2.0:1787758423.234286:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306151105 00000100:00000001:2.0:1787758423.234289:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306150414 00000100:00000001:2.0:1787758423.234291:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306151105 00000100:00000001:2.0:1787758423.234292:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758423.234294:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234296:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758423.234300:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758423/real 1787758423] req@ffff8ffb45c32a00 x1874588659746560/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758423.234308:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758423.234309:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758423.234311:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758423.234313:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758423.234314:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234316:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758423.234317:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758423.234321:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c32a00 x1874588659746560/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758423.234327:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234329:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758423.234331:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234332:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758423.234333:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234335:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758423.234346:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758423.234348:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758423.234351:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758423.234352:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758423.234355:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c32a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746560:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758423.234358:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758423.234359:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758423.234361:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.234363:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758423.234364:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758423.234366:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758423.234368:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746688:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758423.234371:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758423.234373:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758423.234374:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.234376:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758423.234378:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234383:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758423.234384:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102813440 : -123165606738176 : ffff8ffb45c30700) 00000100:00000001:2.0:1787758423.234387:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758423.234397:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758423.234401:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758423.234403:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758423.234404:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234406:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758423.234408:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234410:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306150411 00000100:00000001:2.0:1787758423.234412:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306151105 00000100:00000001:2.0:1787758423.234415:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306150411 00000100:00000001:2.0:1787758423.234416:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306151105 00000100:00000001:2.0:1787758423.234418:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758423.234420:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234421:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758423.234424:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758423/real 1787758423] req@ffff8ffb45c30700 x1874588659746688/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758423.234430:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758423.234431:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758423.234433:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758423.234434:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758423.234436:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234437:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758423.234438:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758423.234441:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30700 x1874588659746688/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758423.234447:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234448:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758423.234450:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234451:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758423.234452:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234454:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758423.234483:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758423.234486:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758423.234490:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758423.234493:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758423.234497:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746688:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758423.234502:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758423.234504:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758423.234505:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.234508:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758423.234510:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758423.234512:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758423.234516:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746816:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758423.234519:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758423.234521:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758423.234523:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758423.234525:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758423.234537:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234542:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758423.234544:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908102814336 : -123165606737280 : ffff8ffb45c30a80) 00000100:00000001:2.0:1787758423.234548:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758423.234560:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758423.234565:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758423.234568:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758423.234570:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234572:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758423.234575:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234579:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306150411 00000100:00000001:2.0:1787758423.234582:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306151105 00000100:00000001:2.0:1787758423.234585:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306150411 00000100:00000001:2.0:1787758423.234588:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306151105 00000100:00000001:2.0:1787758423.234590:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758423.234592:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234594:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758423.234599:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758423/real 1787758423] req@ffff8ffb45c30a80 x1874588659746816/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758423.234616:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758423.234618:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758423.234620:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758423.234622:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758423.234624:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234626:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758423.234627:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758423.234632:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb45c30a80 x1874588659746816/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758423.234640:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234642:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758423.234645:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234647:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758423.234648:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234650:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758423.234662:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758423.234664:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758423.234668:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758423.234670:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758423.234674:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb45c30a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659746816:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758423.234678:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234681:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758423.234682:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758423.234685:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758423.234688:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758423.234690:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758423.234691:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758423.234694:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758423.234695:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758423.234698:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758423.234700:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234703:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758423.234704:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758423.234706:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758423.234708:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758423.234710:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758423.234711:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758423.234713:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758423.234714:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758423.234716:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758423.234718:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234719:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758423.234721:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758423.234722:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758423.234724:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758423.234725:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758423.234726:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758423.234728:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758423.234730:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758423.234731:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758423.234733:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234735:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758423.234739:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758423.234741:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758423.234743:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758423.234745:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758423.809378:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758423.809391:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758423.809395:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758423.809402:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758423.809407:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758423.809414:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758423.809417:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758423.809421:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758423.809425:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758424.001402:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758424.001432:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758424.001444:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758424.001448:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758424.001532:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758424.001539:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758424.001545:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758424.833286:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758424.833294:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758424.833297:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758424.833302:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758424.833307:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758424.833312:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758424.833314:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758424.833317:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758424.833320:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758425.025336:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758425.025345:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758425.025351:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758425.025356:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758425.025399:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758425.025402:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758425.025406:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758425.857351:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758425.857359:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758425.857362:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758425.857368:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758425.857372:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758425.857376:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758425.857379:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758425.857382:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758425.857385:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758426.049346:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758426.049357:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758426.049370:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758426.049377:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758426.049382:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758426.049388:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758426.049394:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758426.049399:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758426.049404:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758426.049409:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758426.049493:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758426.049506:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758426.049511:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758426.881352:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758426.881362:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758426.881365:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758426.881372:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758426.881377:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758426.881382:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758426.881385:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758426.881389:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758426.881392:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758427.073340:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758427.073351:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758427.073360:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758427.073362:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758427.073453:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758427.073461:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758427.073466:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758427.905326:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758427.905338:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758427.905341:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758427.905350:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758427.905355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758427.905361:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758427.905365:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758427.905368:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758427.905372:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758428.097346:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758428.097361:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758428.097365:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758428.097370:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758428.097379:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758428.097382:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758428.097385:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758428.097388:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758428.097394:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758428.097396:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758428.097519:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758428.097528:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758428.097534:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758428.353346:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758428.353356:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:2.0:1787758428.353359:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758428.353363:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758428.353367:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758428.353375:0:18724:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11483 00000100:00080000:2.0:1787758428.353387:0:18724:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758428.353393:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758428.353396:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758428.353400:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758428.353402:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758428.353406:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:2.0:1787758428.353409:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:2.0:1787758428.353411:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758428.353413:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758428.353417:0:18724:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758428.353420:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758428.353422:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758428.353426:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758428.353429:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353431:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:2.0:1787758428.353434:0:18724:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758428.353448:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758428.353450:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353465:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758428.353468:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758428.353473:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353496:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758428.353506:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38000 x1874588659746944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758428.353517:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758428.353521:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353545:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758428.353548:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758428.353552:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:2.0:1787758428.353553:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758428.353555:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758428.353558:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758428.353560:0:18724:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11483 00000100:00080000:2.0:1787758428.353564:0:18724:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758428.353567:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758428.353568:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758428.353571:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758428.353572:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758428.353574:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:2.0:1787758428.353576:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:2.0:1787758428.353578:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758428.353579:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:2.0:1787758428.353590:0:18724:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758428.353592:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758428.353594:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758428.353599:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758428.353601:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353603:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758428.353607:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758428.353610:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353612:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758428.353615:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758428.353617:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353619:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758428.353620:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758428.353622:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353626:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758428.353631:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a38380 x1874588659747072/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758428.353639:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758428.353641:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353644:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758428.353648:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758428.353653:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:2.0:1787758428.353655:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758428.353658:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758428.353661:0:18724:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:2.0:1787758428.353663:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11483 00000100:00080000:2.0:1787758428.353666:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758428.353668:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758428.353670:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758428.353671:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758428.353679:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758428.353680:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000100:00000001:1.0:1787758428.353682:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1787758428.353683:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:2.0:1787758428.353689:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758428.353690:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758428.353701:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758428.353701:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758428.353710:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758428.353716:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758428.353717:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758428.353721:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758428.353729:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1787758428.353731:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758428.353734:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758428.353735:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353744:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758428.353747:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 02000000:00000001:1.0:1787758428.353748:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000008:00000001:2.0:1787758428.353751:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353762:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758428.353763:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758428.353784:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758428.353787:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353792:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758428.353794:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00100000:1.0:1787758428.353797:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659746944:192.168.201.118@tcp:250:kworker.0 02000000:00000001:2.0:1787758428.353797:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353809:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.353817:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:2.0:1787758428.353820:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a39880 x1874588659747200/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1787758428.353853:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1787758428.353855:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758428.353857:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353860:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758428.353863:0:18724:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:1.0:1787758428.353864:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758428.353871:0:18724:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 02000000:00000001:1.0:1787758428.353873:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1787758428.353875:0:18724:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:2.0:1787758428.353877:0:18724:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:2.0:1787758428.353879:0:18724:0:(import.c:485:import_select_connection()) Process entered 02000000:00000001:1.0:1787758428.353882:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1787758428.353882:0:18724:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11483 00000100:00080000:2.0:1787758428.353895:0:18724:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758428.353898:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758428.353899:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:2.0:1787758428.353901:0:18724:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:2.0:1787758428.353902:0:18724:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:2.0:1787758428.353904:0:18724:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:2.0:1787758428.353905:0:18724:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758428.353907:0:18724:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758428.353908:0:18724:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:1.0:1787758428.353912:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:2.0:1787758428.353912:0:18724:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758428.353917:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134301696 : -123165575249920 : ffff8ffb47a38000) 00000100:00000001:2.0:1787758428.353919:0:18724:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758428.353928:0:18724:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:1.0:1787758428.353931:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 02000000:00000001:2.0:1787758428.353933:0:18724:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758428.353937:0:18724:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353938:0:18724:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:2.0:1787758428.353939:0:18724:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:2.0:1787758428.353941:0:18724:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353942:0:18724:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758428.353945:0:18724:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:2.0:1787758428.353946:0:18724:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353948:0:18724:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1787758428.353949:0:18724:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1787758428.353950:0:18724:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353953:0:18724:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758428.353956:0:18724:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb47a3a680 x1874588659747328/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758428.353962:0:18724:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758428.353964:0:18724:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758428.353966:0:18724:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:1.0:1787758428.353979:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758428.353996:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758428.354000:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758428.354003:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354004:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758428.354009:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354013:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306156225 00000100:00000001:1.0:1787758428.354024:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758428.354029:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306156225 00000100:00000001:1.0:1787758428.354033:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00000001:1.0:1787758428.354035:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758428.354038:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354041:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758428.354074:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758428/real 1787758428] req@ffff8ffb47a38000 x1874588659746944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758428.354084:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758428.354086:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758428.354089:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758428.354091:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758428.354092:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354094:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758428.354096:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758428.354100:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38000 x1874588659746944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758428.354105:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354108:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758428.354111:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354112:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758428.354113:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354115:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758428.354116:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758428.354119:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758428.354122:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758428.354123:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758428.354126:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659746944:192.168.201.118@tcp:250:kworker.0 00000100:00000001:1.0:1787758428.354130:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758428.354131:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758428.354133:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758428.354135:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758428.354136:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758428.354138:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758428.354141:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747072:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758428.354143:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758428.354145:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758428.354146:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758428.354147:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758428.354150:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354154:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758428.354155:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134302592 : -123165575249024 : ffff8ffb47a38380) 00000100:00000001:1.0:1787758428.354160:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758428.354169:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758428.354171:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758428.354173:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758428.354174:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354176:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758428.354177:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354212:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306155534 00000100:00000001:1.0:1787758428.354216:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306156225 00000100:00000001:1.0:1787758428.354217:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306155534 00000100:00000001:1.0:1787758428.354219:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306156225 00000100:00000001:1.0:1787758428.354221:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758428.354222:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354223:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758428.354226:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758428/real 1787758428] req@ffff8ffb47a38380 x1874588659747072/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758428.354232:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758428.354233:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758428.354235:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758428.354236:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758428.354237:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354238:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758428.354239:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758428.354242:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a38380 x1874588659747072/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758428.354248:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354249:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758428.354251:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354252:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758428.354253:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354254:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758428.354268:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758428.354270:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758428.354272:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758428.354273:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758428.354275:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a38380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747072:192.168.201.118@tcp:38:kworker.0 00000100:00000001:1.0:1787758428.354278:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354280:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758428.354281:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758428.354284:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758428.354303:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758428.354306:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758428.354309:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758428.354315:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758428.354318:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758428.354324:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758428.354325:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354327:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758428.354329:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758428.354331:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758428.354334:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758428.354336:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758428.354337:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758428.354339:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758428.354341:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758428.354342:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758428.354344:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354347:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354357:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758428.354360:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354363:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758428.354365:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758428.354367:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758428.354370:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758428.354371:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758428.354375:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758428.354377:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758428.354379:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758428.354383:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758428.354388:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758428.354390:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758428.354392:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758428.354395:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758428.354398:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354407:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758428.354409:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134307968 : -123165575243648 : ffff8ffb47a39880) 00000100:00000001:1.0:1787758428.354414:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758428.354430:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758428.354435:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758428.354438:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758428.354440:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354442:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758428.354446:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354450:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306155531 00000100:00000001:1.0:1787758428.354453:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306156225 00000100:00000001:1.0:1787758428.354456:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306155531 00000100:00000001:1.0:1787758428.354459:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306156225 00000100:00000001:1.0:1787758428.354462:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758428.354464:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354466:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758428.354471:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758428/real 1787758428] req@ffff8ffb47a39880 x1874588659747200/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758428.354489:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758428.354490:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758428.354493:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758428.354495:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758428.354497:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354498:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758428.354500:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758428.354505:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a39880 x1874588659747200/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758428.354514:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354516:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758428.354519:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354521:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758428.354523:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354525:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758428.354542:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758428.354545:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758428.354548:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758428.354551:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758428.354555:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a39880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747200:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758428.354559:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758428.354561:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758428.354563:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758428.354566:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:1.0:1787758428.354568:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758428.354570:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758428.354574:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747328:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758428.354577:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:1.0:1787758428.354579:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:1.0:1787758428.354581:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758428.354584:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:1.0:1787758428.354587:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354591:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1787758428.354593:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908134311552 : -123165575240064 : ffff8ffb47a3a680) 00000100:00000001:1.0:1787758428.354598:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:1.0:1787758428.354609:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:1.0:1787758428.354614:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:1.0:1787758428.354616:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758428.354618:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354620:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758428.354623:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354627:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306155531 00000100:00000001:1.0:1787758428.354630:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306156225 00000100:00000001:1.0:1787758428.354633:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306155531 00000100:00000001:1.0:1787758428.354636:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306156225 00000100:00000001:1.0:1787758428.354638:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758428.354641:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354643:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:1.0:1787758428.354647:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758428/real 1787758428] req@ffff8ffb47a3a680 x1874588659747328/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:1.0:1787758428.354656:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1787758428.354657:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:1.0:1787758428.354659:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758428.354661:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:1.0:1787758428.354663:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354665:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758428.354667:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1787758428.354672:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb47a3a680 x1874588659747328/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1787758428.354680:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354682:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:1.0:1787758428.354686:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354688:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1787758428.354690:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354692:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:1.0:1787758428.354727:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:1.0:1787758428.354730:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758428.354733:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758428.354736:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:1.0:1787758428.354739:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb47a3a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747328:192.168.201.118@tcp:8:kworker.0 00000100:00000001:1.0:1787758428.354743:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354745:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758428.354746:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758428.354747:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758428.354749:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758428.354751:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758428.354752:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758428.354756:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758428.354757:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758428.354760:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758428.354762:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354764:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:1.0:1787758428.354765:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1787758428.354766:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:1.0:1787758428.354768:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1787758428.354769:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:1.0:1787758428.354771:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:1.0:1787758428.354773:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1787758428.354774:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:1.0:1787758428.354777:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1787758428.354778:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354780:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.354784:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758428.354785:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.354786:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758428.354788:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758428.929330:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758428.929340:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758428.929343:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758428.929350:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758428.929355:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758428.929360:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.929363:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758428.929366:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758428.929370:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758429.121407:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758429.121421:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758429.121433:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758429.121436:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758429.121502:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758429.121508:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758429.121514:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758429.953353:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758429.953363:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758429.953367:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758429.953376:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758429.953381:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758429.953386:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758429.953389:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758429.953392:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758429.953395:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758430.145347:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758430.145356:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758430.145364:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758430.145366:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758430.145446:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758430.145454:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758430.145460:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758430.977305:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758430.977315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758430.977318:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758430.977324:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758430.977329:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758430.977334:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758430.977338:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758430.977341:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758430.977344:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758431.169342:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758431.169352:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758431.169362:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758431.169364:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758431.169435:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758431.169440:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758431.169445:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758432.001307:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758432.001319:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758432.001324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758432.001333:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758432.001339:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758432.001345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758432.001350:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758432.001354:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758432.001359:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758432.193469:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758432.193476:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758432.193481:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758432.193482:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758432.193544:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758432.193549:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758432.193554:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758433.025395:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758433.025405:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758433.025407:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758433.025413:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758433.025417:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758433.025422:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758433.025424:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758433.025426:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758433.025429:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758433.217375:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758433.217385:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758433.217392:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758433.217394:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758433.217420:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758433.217424:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758433.217428:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758433.473404:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758433.473424:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758433.473429:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758433.473435:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758433.473443:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758433.473457:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11488 00000100:00080000:1.0:1787758433.473476:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758433.473481:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758433.473483:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758433.473489:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758433.473492:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758433.473496:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758433.473499:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758433.473501:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758433.473502:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758433.473508:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758433.473511:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.473515:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758433.473519:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758433.473523:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473525:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758433.473529:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.473548:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758433.473550:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473554:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758433.473560:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758433.473566:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473584:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758433.473596:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534700 x1874588659747456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758433.473609:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758433.473614:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473641:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758433.473645:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758433.473649:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758433.473651:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758433.473653:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758433.473657:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758433.473659:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11488 00000100:00080000:1.0:1787758433.473663:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758433.473666:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758433.473667:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758433.473670:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758433.473671:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758433.473673:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758433.473676:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758433.473677:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758433.473679:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758433.473682:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758433.473684:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.473685:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758433.473693:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758433.473695:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473697:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758433.473703:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758433.473706:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473709:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.473713:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758433.473715:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473718:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758433.473720:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758433.473723:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473733:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758433.473744:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a535c00 x1874588659747584/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758433.473754:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1787758433.473759:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758433.473768:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.473769:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758433.473776:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758433.473782:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758433.473785:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.473785:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758433.473801:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 00000100:00080000:1.0:1787758433.473802:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758433.473812:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 02000000:00000001:2.0:1787758433.473812:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758433.473826:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 02000000:00000001:2.0:1787758433.473827:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758433.473833:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758433.473841:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00100000:2.0:1787758433.473843:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659747456:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1787758433.473853:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11488 00000100:00000001:2.0:1787758433.473863:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00080000:1.0:1787758433.473871:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 02000000:00000001:2.0:1787758433.473873:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1787758433.473883:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 02000000:00000001:2.0:1787758433.473883:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473895:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 02000000:00000001:2.0:1787758433.473896:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000020:00000001:1.0:1787758433.473906:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758433.473908:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758433.473910:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758433.473913:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 02000000:00000001:2.0:1787758433.473913:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.473928:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758433.473932:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758433.473941:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758433.473945:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.473947:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758433.473950:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 02000000:00000001:1.0:1787758433.473952:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758433.473980:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.473982:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179375872 : -123165530175744 : ffff8ffb4a534700) 00000100:00000001:1.0:1787758433.473984:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758433.473992:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758433.473994:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000008:00000001:1.0:1787758433.474001:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.474005:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.474014:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758433.474016:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.474018:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758433.474021:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758433.474023:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.474030:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758433.474037:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659747712/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758433.474046:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758433.474049:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.474053:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000800:00000100:2.0:1787758433.474061:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1787758433.474064:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758433.474071:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758433.474073:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758433.474076:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758433.474083:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787758433.474083:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00080000:1.0:1787758433.474098:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11488 00000100:00000001:2.0:1787758433.474100:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758433.474105:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758433.474106:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758433.474121:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:2.0:1787758433.474121:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:1.0:1787758433.474137:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758433.474142:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758433.474144:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758433.474148:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758433.474156:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:2.0:1787758433.474157:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306161345 00000020:00000001:1.0:1787758433.474166:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758433.474172:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758433.474174:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758433.474178:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00080000:1.0:1787758433.474236:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758433.474243:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.474246:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758433.474253:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758433.474258:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306161346 02000000:00000001:1.0:1787758433.474259:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.474267:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758433.474274:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00000001:2.0:1787758433.474277:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000008:00000001:1.0:1787758433.474280:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758433.474289:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.474299:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758433.474303:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474305:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1787758433.474309:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758433.474314:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758433.474318:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758433.474322:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474328:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758433.474334:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758433.474363:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758433/real 1787758433] req@ffff8ffb4a534700 x1874588659747456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758433.474367:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659747840/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000001:2.0:1787758433.474456:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787758433.474461:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000400:00000001:2.0:1787758433.474461:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758433.474477:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758433.474479:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474484:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758433.474488:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758433.474490:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474494:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758433.474499:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758433.474512:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534700 x1874588659747456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758433.474524:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474530:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758433.474538:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474541:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758433.474544:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474547:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758433.474551:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758433.474556:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758433.474563:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758433.474566:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758433.474570:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659747456:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758433.474576:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474579:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758433.474581:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758433.474585:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758433.474590:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758433.474594:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758433.474597:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758433.474602:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758433.474605:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758433.474609:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758433.474611:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474614:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474622:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758433.474625:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474628:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758433.474631:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758433.474633:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758433.474635:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758433.474637:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758433.474641:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758433.474653:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758433.474656:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758433.474660:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747584:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758433.474664:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758433.474667:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758433.474669:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758433.474672:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758433.474676:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474684:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758433.474686:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179381248 : -123165530170368 : ffff8ffb4a535c00) 00000100:00000001:2.0:1787758433.474691:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758433.474721:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758433.474727:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758433.474730:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758433.474732:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474734:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758433.474738:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474743:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306160654 00000100:00000001:2.0:1787758433.474747:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306161345 00000100:00000001:2.0:1787758433.474751:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306160654 00000100:00000001:2.0:1787758433.474754:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306161346 00000100:00000001:2.0:1787758433.474756:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758433.474759:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474762:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758433.474767:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758433/real 1787758433] req@ffff8ffb4a535c00 x1874588659747584/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758433.474778:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758433.474780:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758433.474783:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758433.474785:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758433.474787:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474789:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758433.474791:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758433.474796:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a535c00 x1874588659747584/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758433.474804:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474807:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758433.474810:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474812:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758433.474813:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.474815:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758433.474832:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758433.474835:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758433.474839:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758433.474841:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758433.474845:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a535c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747584:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758433.474849:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758433.474852:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758433.474854:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758433.474857:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758433.474859:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758433.474861:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758433.474865:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758433.474873:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758433.474875:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758433.474877:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758433.474879:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758433.474882:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474888:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758433.474890:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787758433.474895:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758433.474907:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758433.474911:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758433.474914:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758433.474916:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474918:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758433.474921:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474925:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306160651 00000100:00000001:2.0:1787758433.474928:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306161345 00000100:00000001:2.0:1787758433.474931:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306160651 00000100:00000001:2.0:1787758433.474934:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306161346 00000100:00000001:2.0:1787758433.474936:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758433.474939:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.474941:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758433.474945:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758433/real 1787758433] req@ffff8ffb4a537800 x1874588659747712/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758433.474991:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758433.474993:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758433.474996:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758433.474998:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758433.475000:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475003:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758433.475004:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758433.475010:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659747712/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758433.475027:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475030:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758433.475033:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475035:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758433.475037:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475039:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758433.475053:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758433.475056:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758433.475060:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758433.475062:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758433.475066:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747712:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758433.475071:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758433.475073:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758433.475075:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758433.475078:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758433.475081:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758433.475084:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758433.475087:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747840:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758433.475091:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758433.475094:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758433.475096:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758433.475098:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758433.475102:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.475108:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758433.475110:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787758433.475115:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758433.475130:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758433.475136:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758433.475139:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758433.475141:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.475143:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758433.475147:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.475151:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306160651 00000100:00000001:2.0:1787758433.475154:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306161345 00000100:00000001:2.0:1787758433.475157:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306160651 00000100:00000001:2.0:1787758433.475160:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306161346 00000100:00000001:2.0:1787758433.475163:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758433.475165:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.475167:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758433.475172:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758433/real 1787758433] req@ffff8ffb4a537b80 x1874588659747840/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758433.475181:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758433.475182:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758433.475224:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758433.475227:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758433.475230:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475233:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758433.475236:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758433.475246:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659747840/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758433.475261:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475265:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758433.475270:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475273:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758433.475276:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475279:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758433.475301:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758433.475306:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758433.475314:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758433.475317:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758433.475322:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659747840:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758433.475327:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475330:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758433.475332:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758433.475334:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758433.475338:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758433.475340:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758433.475342:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758433.475346:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758433.475357:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758433.475360:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758433.475362:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475364:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758433.475365:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758433.475367:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758433.475369:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758433.475371:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758433.475372:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758433.475374:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758433.475375:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758433.475377:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758433.475379:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475380:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758433.475382:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758433.475383:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758433.475385:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758433.475386:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758433.475387:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758433.475390:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758433.475392:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758433.475393:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758433.475395:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475397:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758433.475401:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758433.475403:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758433.475406:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758433.475408:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758434.049282:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758434.049288:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758434.049290:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758434.049295:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758434.049299:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758434.049302:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758434.049304:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758434.049307:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758434.049309:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758434.241340:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758434.241349:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758434.241362:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758434.241366:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758434.241408:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758434.241413:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758434.241416:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758435.073263:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758435.073268:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758435.073269:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758435.073273:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758435.073275:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758435.073279:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758435.073280:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758435.073282:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758435.073284:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758435.265252:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758435.265257:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758435.265263:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758435.265265:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758435.265267:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758435.265270:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758435.265272:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758435.265274:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758435.265275:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758435.265277:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758435.265316:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758435.265320:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758435.265323:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758436.097313:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758436.097322:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758436.097324:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758436.097330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758436.097334:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758436.097339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758436.097341:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758436.097343:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758436.097346:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758436.289295:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758436.289301:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758436.289306:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758436.289307:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1787758436.289401:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1787758436.289404:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:2.0:1787758436.289407:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758437.121302:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758437.121311:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758437.121314:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758437.121320:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758437.121324:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758437.121329:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758437.121332:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758437.121335:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758437.121338:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758437.313304:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758437.313318:0:178343:0:(ldlm_request.c:2324:ldlm_cancel_lru()) Process entered 00010000:00000001:0.0:1787758437.313322:0:178343:0:(ldlm_request.c:2112:ldlm_prepare_lru_list()) Process entered 00010000:00000001:0.0:1787758437.313339:0:178343:0:(ldlm_request.c:2284:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758437.313347:0:178343:0:(ldlm_lockd.c:2241:ldlm_bl_to_thread()) Process entered 00010000:00000001:0.0:1787758437.313350:0:178343:0:(ldlm_lockd.c:2244:ldlm_bl_to_thread()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758437.313354:0:178343:0:(ldlm_request.c:2334:ldlm_cancel_lru()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758437.313357:0:178343:0:(ldlm_pool.c:496:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758437.313364:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758437.313366:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1787758437.313391:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1787758437.313394:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1787758437.313396:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758438.145315:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758438.145322:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758438.145325:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758438.145330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758438.145335:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758438.145339:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758438.145343:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758438.145346:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758438.145349:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758438.337318:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758438.337326:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758438.337333:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758438.337335:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758438.337407:0:152732:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758438.337413:0:152732:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758438.337417:0:152732:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758438.593300:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) 489ffc11-9b85-4cfb-9796-86e778a35647->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758438.593313:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00000001:1.0:1787758438.593317:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758438.593321:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758438.593327:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758438.593336:0:87148:0:(import.c:510:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 11493 00000100:00080000:1.0:1787758438.593349:0:87148:0:(import.c:589:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758438.593354:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758438.593356:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758438.593361:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758438.593364:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758438.593368:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000020:00000001:1.0:1787758438.593371:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908173957120 : -123165535594496 : ffff8ffb4a009800) 00000100:00000001:1.0:1787758438.593374:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758438.593376:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758438.593382:0:87148:0:(import.c:624:import_select_connection()) MGC192.168.201.118@tcp: import ffff8ffb4a00e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758438.593385:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758438.593388:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758438.593392:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758438.593397:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593400:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00000001:1.0:1787758438.593404:0:87148:0:(obd_class.h:895:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758438.593420:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758438.593423:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593430:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758438.593434:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:1.0:1787758438.593441:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593460:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758438.593477:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a536300 x1874588659747968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:1.0:1787758438.593517:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758438.593523:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593549:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758438.593555:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758438.593562:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000001:1.0:1787758438.593566:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758438.593569:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758438.593575:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758438.593580:0:87148:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11493 00000100:00080000:1.0:1787758438.593589:0:87148:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:1.0:1787758438.593594:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758438.593597:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000020:00000001:1.0:1787758438.593618:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758438.593621:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758438.593624:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000020:00000001:1.0:1787758438.593629:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908300746752 : -123165408804864 : ffff8ffb518f4000) 00000100:00000001:1.0:1787758438.593633:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758438.593635:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758438.593642:0:87148:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ffb481c8000: import ffff8ffb49318000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758438.593647:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758438.593650:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758438.593658:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758438.593662:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593665:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000008:00100000:1.0:1787758438.593671:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758438.593677:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593682:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.593690:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1787758438.593692:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758438.593700:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.593701:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758438.593707:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:1.0:1787758438.593711:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787758438.593712:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 02000000:00000001:1.0:1787758438.593720:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758438.593720:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758438.593730:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758438.593731:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758438.593738:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758438.593743:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593748:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758438.593753:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659747968:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758438.593763:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 00000100:00100000:1.0:1787758438.593767:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a534a80 x1874588659748096/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:2.0:1787758438.593770:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758438.593777:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758438.593786:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1787758438.593794:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758438.593798:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593803:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758438.593807:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 02000000:00000001:2.0:1787758438.593812:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758438.593813:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000001:1.0:1787758438.593821:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758438.593824:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758438.593829:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00080000:1.0:1787758438.593832:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11493 00000100:00000001:2.0:1787758438.593838:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00080000:1.0:1787758438.593840:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758438.593845:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179383040 : -123165530168576 : ffff8ffb4a536300) 00000100:00000001:1.0:1787758438.593848:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758438.593854:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00000001:2.0:1787758438.593855:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000020:00000001:1.0:1787758438.593859:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758438.593891:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758438.593896:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000020:00000001:1.0:1787758438.593900:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410060800 : -123165299490816 : ffff8ffb58134000) 00000100:00000001:1.0:1787758438.593903:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758438.593906:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00080000:1.0:1787758438.593911:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ffb481c8000: import ffff8ffb49399000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:1.0:1787758438.593915:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758438.593918:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:1.0:1787758438.593923:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1787758438.593926:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593929:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000800:00000100:2.0:1787758438.593936:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1787758438.593937:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758438.593943:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593946:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.593951:0:2367:0:(events.c:34:request_out_callback()) Process entered 02000000:00000001:1.0:1787758438.593955:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 00000100:00000001:2.0:1787758438.593960:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 02000000:00000001:1.0:1787758438.593961:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593969:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 00000100:00000001:2.0:1787758438.593969:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758438.593975:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758438.593977:0:2367:0:(events.c:68:request_out_callback()) Process leaving 02000000:00000001:1.0:1787758438.593980:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.593987:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.593991:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.593996:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 2 is 4306166465 00000100:00100000:1.0:1787758438.594006:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537b80 x1874588659748224/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758438.594014:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 2 00000100:00000001:1.0:1787758438.594021:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1787758438.594028:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594028:0:2367:0:(niobuf.c:663:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS new deadline estimation for cpu 3 is 4306166465 00000100:00000001:1.0:1787758438.594041:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594042:0:2367:0:(niobuf.c:678:kick_cpu_latency()) MGC192.168.201.118@tcp: PM QoS request now active for cpu 3 00000100:00080000:1.0:1787758438.594046:0:87148:0:(pinger.c:229:ptlrpc_pinger_process_import()) efafdbc1-e54a-4d99-899c-0cf2da167817->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758438.594052:0:87148:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000001:1.0:1787758438.594054:0:87148:0:(import.c:701:ptlrpc_connect_import_locked()) Process entered 00000100:00080000:1.0:1787758438.594055:0:87148:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000001:1.0:1787758438.594058:0:87148:0:(import.c:485:import_select_connection()) Process entered 00000100:00000001:2.0:1787758438.594061:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00080000:1.0:1787758438.594063:0:87148:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: connect to NID 192.168.201.118@tcp last attempt 11493 00000100:00000001:2.0:1787758438.594070:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:1.0:1787758438.594074:0:87148:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: tried all connections, increasing latency to 20s 00000100:00000001:2.0:1787758438.594077:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00000001:1.0:1787758438.594081:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000100:00000001:1.0:1787758438.594085:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000100:00100000:2.0:1787758438.594092:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758438/real 1787758438] req@ffff8ffb4a536300 x1874588659747968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000001:1.0:1787758438.594093:0:87148:0:(genops.c:792:class_conn2export()) Process entered 00000020:00000001:1.0:1787758438.594099:0:87148:0:(lustre_handles.c:130:class_handle2object()) Process entered 00000020:00000001:1.0:1787758438.594105:0:87148:0:(lustre_handles.c:155:class_handle2object()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000020:00000001:1.0:1787758438.594114:0:87148:0:(genops.c:806:class_conn2export()) Process leaving (rc=18446620908410044416 : -123165299507200 : ffff8ffb58130000) 00000100:00000001:1.0:1787758438.594124:0:87148:0:(connection.c:156:ptlrpc_connection_addref()) Process entered 00000400:00000001:2.0:1787758438.594127:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:1.0:1787758438.594128:0:87148:0:(connection.c:163:ptlrpc_connection_addref()) Process leaving (rc=18446620908089445184 : -123165620106432 : ffff8ffb44f70b40) 00000400:00000001:2.0:1787758438.594133:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00080000:1.0:1787758438.594140:0:87148:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ffb481c8000: import ffff8ffb45643000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00000001:2.0:1787758438.594143:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1787758438.594146:0:87148:0:(import.c:628:import_select_connection()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594150:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 02000000:00000001:1.0:1787758438.594152:0:87148:0:(sec.c:1579:sptlrpc_import_sec_adapt()) Process entered 00000100:00000001:2.0:1787758438.594156:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1787758438.594159:0:87148:0:(sec.c:1619:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758438.594162:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000001:1.0:1787758438.594168:0:87148:0:(sec.c:1647:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594169:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758438.594176:0:87148:0:(obd_class.h:888:obd_reconnect()) Process entered 00000100:00100000:2.0:1787758438.594222:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a536300 x1874588659747968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758438.594227:0:87148:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000008:00000001:1.0:1787758438.594235:0:87148:0:(osc_request.c:3833:osc_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.594237:0:87148:0:(obd_class.h:902:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1787758438.594243:0:87148:0:(sec.c:2038:sptlrpc_sepol_get()) Process entered 02000000:00000001:1.0:1787758438.594247:0:87148:0:(sec.c:2048:sptlrpc_sepol_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594250:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758438.594252:0:87148:0:(client.c:858:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1787758438.594258:0:87148:0:(sec.c:433:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:2.0:1787758438.594259:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 02000000:00000001:1.0:1787758438.594262:0:87148:0:(sec.c:457:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594268:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758438.594273:0:87148:0:(client.c:938:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594274:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758438.594279:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594284:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00100000:1.0:1787758438.594290:0:87148:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ffb4a537800 x1874588659748352/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1787758438.594290:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758438.594302:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb4a00e000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758438.594319:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00000001:1.0:1787758438.594321:0:87148:0:(jobid.c:959:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1787758438.594325:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1787758438.594327:0:87148:0:(jobid.c:999:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758438.594336:0:87148:0:(import.c:833:ptlrpc_connect_import_locked()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758438.594338:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a536300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:489ffc11-9b85-4cfb-9796-86e778a35647:2367:1874588659747968:192.168.201.118@tcp:250:kworker.0 00000100:00000001:2.0:1787758438.594347:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594350:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758438.594352:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758438.594355:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758438.594361:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758438.594365:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758438.594368:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758438.594373:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758438.594375:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758438.594379:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758438.594380:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594383:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594391:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758438.594393:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594396:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758438.594398:0:2367:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1787758438.594400:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758438.594403:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758438.594404:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758438.594407:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758438.594410:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758438.594412:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758438.594416:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659748096:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758438.594420:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758438.594422:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758438.594424:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758438.594426:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758438.594430:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594436:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758438.594438:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179376768 : -123165530174848 : ffff8ffb4a534a80) 00000100:00000001:2.0:1787758438.594444:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758438.594459:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758438.594464:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758438.594467:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758438.594469:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594470:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758438.594474:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594478:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306165774 00000100:00000001:2.0:1787758438.594482:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306166465 00000100:00000001:2.0:1787758438.594485:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306165774 00000100:00000001:2.0:1787758438.594488:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-MDT0000-mdc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306166465 00000100:00000001:2.0:1787758438.594490:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758438.594492:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594494:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758438.594499:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758438/real 1787758438] req@ffff8ffb4a534a80 x1874588659748096/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758438.594509:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758438.594510:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758438.594512:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758438.594514:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758438.594517:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594519:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758438.594521:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758438.594525:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a534a80 x1874588659748096/t0(0) o38->lustre-MDT0000-mdc-ffff8ffb481c8000@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758438.594533:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594535:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758438.594538:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594540:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758438.594541:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594543:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758438.594557:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758438.594559:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49318000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758438.594563:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758438.594565:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758438.594568:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a534a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659748096:192.168.201.118@tcp:38:kworker.0 00000100:00000001:2.0:1787758438.594573:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758438.594574:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758438.594576:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758438.594579:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758438.594581:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758438.594583:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758438.594587:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659748224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758438.594590:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758438.594591:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758438.594593:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758438.594595:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758438.594598:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594603:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758438.594605:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179389312 : -123165530162304 : ffff8ffb4a537b80) 00000100:00000001:2.0:1787758438.594609:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758438.594620:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758438.594624:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758438.594626:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758438.594628:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594630:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758438.594633:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594637:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306165771 00000100:00000001:2.0:1787758438.594640:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306166465 00000100:00000001:2.0:1787758438.594643:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306165771 00000100:00000001:2.0:1787758438.594645:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0000-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306166465 00000100:00000001:2.0:1787758438.594648:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758438.594650:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594652:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758438.594656:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758438/real 1787758438] req@ffff8ffb4a537b80 x1874588659748224/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758438.594664:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758438.594665:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758438.594667:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758438.594669:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758438.594670:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594672:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758438.594674:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758438.594677:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537b80 x1874588659748224/t0(0) o8->lustre-OST0000-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758438.594685:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594687:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758438.594690:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594692:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758438.594694:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594696:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758438.594707:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758438.594709:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb49399000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758438.594713:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758438.594715:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758438.594719:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659748224:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758438.594723:0:2367:0:(client.c:1813:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1787758438.594725:0:2367:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:2.0:1787758438.594726:0:2367:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758438.594728:0:2367:0:(sec.c:709:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:2.0:1787758438.594730:0:2367:0:(sec.c:740:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1787758438.594732:0:2367:0:(sec.c:784:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1787758438.594735:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659748352:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758438.594738:0:2367:0:(niobuf.c:866:ptl_send_rpc()) Process entered 02000000:00000001:2.0:1787758438.594740:0:2367:0:(sec.c:1080:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1787758438.594742:0:2367:0:(sec.c:1117:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1787758438.594743:0:2367:0:(sec.c:1855:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000001:2.0:1787758438.594747:0:2367:0:(sec.c:1865:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594751:0:2367:0:(client.c:3275:ptlrpc_request_addref()) Process entered 00000100:00000001:2.0:1787758438.594753:0:2367:0:(client.c:3277:ptlrpc_request_addref()) Process leaving (rc=18446620908179388416 : -123165530163200 : ffff8ffb4a537800) 00000100:00000001:2.0:1787758438.594757:0:2367:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000100:2.0:1787758438.594766:0:2367:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00000001:2.0:1787758438.594770:0:2367:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1787758438.594773:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758438.594775:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594777:0:2367:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1787758438.594779:0:2367:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594783:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 2 is 4306165771 00000100:00000001:2.0:1787758438.594786:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 2, keep current deadline 4306166465 00000100:00000001:2.0:1787758438.594789:0:2367:0:(niobuf.c:663:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS new deadline estimation for cpu 3 is 4306165771 00000100:00000001:2.0:1787758438.594791:0:2367:0:(niobuf.c:692:kick_cpu_latency()) lustre-OST0001-osc-ffff8ffb481c8000: PM QoS request active for cpu 3, keep current deadline 4306166465 00000100:00000001:2.0:1787758438.594794:0:2367:0:(niobuf.c:1135:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1787758438.594796:0:2367:0:(client.c:1941:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594798:0:2367:0:(client.c:2485:ptlrpc_expire_one_request()) Process entered 00000100:00100000:2.0:1787758438.594803:0:2367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758438/real 1787758438] req@ffff8ffb4a537800 x1874588659748352/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000400:00000001:2.0:1787758438.594811:0:2367:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:2.0:1787758438.594813:0:2367:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:2.0:1787758438.594815:0:2367:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1787758438.594817:0:2367:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1787758438.594818:0:2367:0:(client.c:3001:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594820:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758438.594822:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:2.0:1787758438.594826:0:2367:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ffb4a537800 x1874588659748352/t0(0) o8->lustre-OST0001-osc-ffff8ffb481c8000@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1787758438.594834:0:2367:0:(client.c:2539:ptlrpc_expire_one_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594836:0:2367:0:(client.c:2124:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000001:2.0:1787758438.594839:0:2367:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594841:0:2367:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1787758438.594842:0:2367:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594844:0:2367:0:(import.c:1057:ptlrpc_connect_interpret()) Process entered 00000100:00000001:2.0:1787758438.594855:0:2367:0:(import.c:1092:ptlrpc_connect_interpret()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00080000:2.0:1787758438.594858:0:2367:0:(import.c:65:import_set_state_nolock()) ffff8ffb45643000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758438.594861:0:2367:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00000001:2.0:1787758438.594884:0:2367:0:(import.c:1517:ptlrpc_connect_interpret()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:2.0:1787758438.594889:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ffb4a537800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:efafdbc1-e54a-4d99-899c-0cf2da167817:2367:1874588659748352:192.168.201.118@tcp:8:kworker.0 00000100:00000001:2.0:1787758438.594893:0:2367:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594895:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758438.594897:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758438.594899:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758438.594902:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758438.594904:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758438.594905:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758438.594909:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758438.594910:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758438.594912:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758438.594915:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594917:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758438.594918:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758438.594920:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758438.594922:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758438.594923:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758438.594925:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758438.594927:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758438.594928:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758438.594930:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758438.594932:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594933:0:2367:0:(client.c:2867:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1787758438.594934:0:2367:0:(client.c:2793:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1787758438.594936:0:2367:0:(sec.c:1880:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000001:2.0:1787758438.594937:0:2367:0:(sec.c:1896:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1787758438.594938:0:2367:0:(genops.c:1217:class_import_put()) Process entered 00000020:00000001:2.0:1787758438.594940:0:2367:0:(genops.c:1230:class_import_put()) Process leaving 02000000:00000001:2.0:1787758438.594941:0:2367:0:(sec.c:473:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1787758438.594942:0:2367:0:(sec.c:490:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1787758438.594944:0:2367:0:(client.c:2849:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1787758438.594945:0:2367:0:(client.c:2908:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594947:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1787758438.594950:0:2367:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1787758438.594952:0:2367:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1787758438.594954:0:2367:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1787758438.594956:0:2367:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1787758439.169299:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758439.169315:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758439.169320:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758439.169330:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758439.169338:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758439.169345:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758439.169351:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758439.169357:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758439.169362:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758439.361344:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758439.361350:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758439.361356:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758439.361357:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758439.361403:0:152733:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758439.361408:0:152733:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758439.361412:0:152733:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758440.193253:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758440.193261:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758440.193263:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758440.193268:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758440.193271:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758440.193274:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758440.193276:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758440.193278:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758440.193281:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758440.385329:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758440.385340:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758440.385349:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758440.385352:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1787758440.385445:0:178000:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1787758440.385454:0:178000:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1787758440.385460:0:178000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000001:1.0:1787758441.217277:0:177997:0:(client.c:2567:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1787758441.217284:0:177997:0:(client.c:2639:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1787758441.217286:0:177997:0:(client.c:2667:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1787758441.217290:0:177997:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ffb47998b40 going to sleep for 0 seconds 00000100:00000001:1.0:1787758441.217293:0:177997:0:(client.c:1990:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1787758441.217296:0:177997:0:(client.c:2993:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758441.217298:0:177997:0:(client.c:1377:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1787758441.217300:0:177997:0:(client.c:1436:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1787758441.217302:0:177997:0:(client.c:2465:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758441.409290:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758441.409296:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1787758441.409301:0:178343:0:(ldlm_pool.c:457:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:0.0:1787758441.409302:0:178343:0:(ldlm_pool.c:461:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1787758441.409329:0:171581:0:(genops.c:1895:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1787758441.409333:0:171581:0:(genops.c:1909:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:1.0:1787758441.409336:0:171581:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 132719 lines, 132719 kept, 0 dropped, 0 bad.